remove incorrect usage of curls Multi API#185
Open
flokaiser wants to merge 1 commit intominio:mainfrom
Open
Conversation
There are several issues introduced by minio#29 When running an application that is rarely restarted, that links to this sdk I encountered hangs inside of the select() call inside of http.c multiple times, that required a restart of the application. Looking into the man pages of curl, it is quite obvious, that the usage of select is a bad idea here. On one hand, the requests.fdset() call may return no filedescriptors, if there is currently nothing to wait for. (https://curl.se/libcurl/c/curl_multi_fdset.html) when fdset() returns no filedescriptors, maxfd is set to -1. this leads to a select call, that has nfds set to 0 and a disabled timeout. -> this leads to an infinite select() until either the program is terminated or a signal is received. On the other hand, if your application is running for some time, new sockets may have filedescriptors larger than FD_SETSIZE (1024), according to the docs (https://curl.se/libcurl/c/curl_multi_fdset.html), fdset does not return any filedescriptors in that case, which leads to the infinite select() again. There are two possible solutions to this: 1. waiting for this PR to be merged: jpbarrette/curlpp#173 - Use the poll() or wait() call from there 2. revert to the Easy API again. - If I understand it correctly the only reason for using the Multi API was to be able to abort transfers inside of the ResponseCallback, but that was implemented incorrectly anyway. Currently the request is removed by requests.remove(request) inside of the ResponseCallback, which is forbidden by: 'It is fine to remove a handle at any time during a transfer, just not from within any libcurl callback function.' (https://curl.se/libcurl/c/curl_multi_remove_handle.html) Due to these reasons I propose to revert back to the Easy API. To keep the possibility to abort a running request, CURL_WRITEFUNC_ERROR is returned at the corresponding locations inside of the ResponseCallback() function.
balamurugana
requested changes
Jan 6, 2025
Member
balamurugana
left a comment
There was a problem hiding this comment.
@flokaiser The requirement is described https://github.com/minio/minio-cpp/pull/29/files#r892559012. That's the reason curlpp::Multi is used. If it is curlpp issue, there is no issue using low level libcurl C APIs directly. Feel free to update the PR accordingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There are several issues introduced by #29
When running an application that is rarely restarted, that links to this sdk I encountered hangs inside of the select() call inside of http.c multiple times, that required a restart of the application.
Looking into the man pages of curl, it is quite obvious, that the usage of select is a bad idea here.
On one hand, the requests.fdset() call may return no filedescriptors, if there is currently nothing to wait for. (https://curl.se/libcurl/c/curl_multi_fdset.html)
when fdset() returns no filedescriptors, maxfd is set to -1. this leads to a select call, that has nfds set to 0 and a disabled timeout. -> this leads to an infinite select() until either the program is terminated or a signal is received.
On the other hand, if your application is running for some time, new sockets may have filedescriptors larger than FD_SETSIZE (1024), according to the docs (https://curl.se/libcurl/c/curl_multi_fdset.html), fdset does not return any filedescriptors in that case, which leads to the infinite select() again.
There are two possible solutions to this:
Due to these reasons I propose to revert back to the Easy API. To keep the possibility to abort a running request, CURL_WRITEFUNC_ERROR is returned at the corresponding locations inside of the ResponseCallback() function.