tuning for high high concurrency when most urls are invalid

[crazysvn]

I have like 100M urls to visit, but 70% of them are invalid. how to tune it to get high speed? right now it performance at only 300k/s, although if all the urls are valid, the speed can achieve 3M/s.
I want 5M/s at least(when 70% of the urls are invalid).. when I use apache httpclient4 with multi threads, the speed can be 3M/s. I expected ACH to be more effective.
what should I do to improve?

public AsyncBatchHttpClient() {
DefaultAsyncHttpClientConfig.Builder configBuilder =
new DefaultAsyncHttpClientConfig.Builder()
.setHandshakeTimeout(1000)
.setConnectTimeout(Duration.ofSeconds(1))
.setConnectionTtl(Duration.ofSeconds(15))
.setRequestTimeout(Duration.ofSeconds(10))
.setReadTimeout(Duration.ofSeconds(10))
////.setShutdownQuietPeriod(Duration.ofSeconds(15))
.setMaxConnections(100000)
.setIoThreadsCount(10000)
.setPooledConnectionIdleTimeout(Duration.ofSeconds(60))
.setSoLinger(0)
.setTcpNoDelay(true)
.setMaxConnectionsPerHost(5)
.setFollowRedirect(true)
.setMaxRedirects(3)
.setMaxRequestRetry(1)
.setKeepAlive(false)
.setUseInsecureTrustManager(true)
.setDisableHttpsEndpointIdentificationAlgorithm(true);

    this.httpClient = Dsl.asyncHttpClient(configBuilder);
    
    System.out.println("MaxConnections: "+httpClient.getConfig().getMaxConnections());
}

[hyperxpro]

Can you please show a snippet for the Apache client too? Also, are you running this in a cluster or a single machine?

[leedyx]

.setMaxRequestRetry(0)

threads count should be 2 * core_of_cpu

[ytatichno]

Maybe you could record a profile of this run and share.