Unable to submit research

[Foxpire]

Hello. Some researches cannot be submitted. I have narrowed this problem to be problem with mobile browsers, since desktop(pc) works always, some researches can be submit when using mobile browsers desktop mode.

I am using Android 8 and Chrome. Same error seems to happen many other devices and browsers. I have already give advice to use chrome since it seems to have best support. I keep asking from our community about device OS's and browser's to gather more info.

Error occurs only when prize parameter is " [n] prize". Just sometimes this happens, so it could be event related research. Happens only when you need to manually tell quantity of items to get as a prize. My guess is that it could be related to event handling.

[bilde2910]

Hi, and thanks for reporting this issue. I have tested this extensively on v1.1.3 and have been unable to reproduce this with Chrome and Firefox on Android 9, which is the only compatible device I currently have available. The officially supported browsers for FreeField on Android are Chrome and Firefox. I also doubt that this is related to event research as the way it is treated internally is exactly the same as other research tasks.

This looks to be a pretty hard to resolve issue, and I will need quite a bit of information in order to troubleshoot it. Could you please tell me the following things about cases when this occurs?

• Most importantly, what happens when research submission fails - do you get an error message when you try to submit research, does it submit but not show up on the map afterwards, or does nothing happen at all when you tap the Report button?
• Are there any instances at all where reporting this type of research from a mobile browser does work (not using desktop mode) when the same task fails on another device?
• If reporting fails for one device, does it always fail for that device as long as they do not use desktop mode?
• Is there anything in common between a lot of the affected devices, as far you can tell? Things like device manufacturer, language, mobile network provider used etc.
• For one of the devices, can you test if reporting works on WiFi vs. mobile data?

Also, since you mentioned you host your own server:

• Are there any PHP notices/warnings/errors/fatal errors in your server's error logs that appear around the time users try to submit research?
• Is the server running PHP 7, or the older, outdated PHP 5?
• Are you using another HTTP and SQL daemon combination than Apache + MySQL?
• Are you using a web application firewall (WAF), load balancer or reverse proxy (CloudFlare-type) in front of your server, or have you configured your server to block some connections based on the User-Agent or IP address of connecting clients?
• Do you use HTTPS?

I might have to ask for some additional information later, but for now, whatever you can provide of what I asked above will be very helpful in trying to figure out exactly what is going on here.

And lastly, sorry for asking this, it might seem like a stupid question but I need to know this for verifying the issue - when reporting research that offers an "[n] prize" reward, can you confirm that the additional box labeled "Quantity" underneath the selected reward is correctly filled with a value and not left blank when clicking Report?

Thanks for the report and information so far.

[Foxpire]

Here is video attached what happens when submit does not go trough. - Most importantly, what happens when research submission fails - do you get an error message when you try to submit research, does it submit but not show up on the map afterwards, or does nothing happen at all when you tap the Report button? - No error message, just unable to proceed. - - Are there any instances at all where reporting this type of research from a mobile browser *does* work (not using desktop mode) when the same task fails on another device? - No. - - If reporting fails for one device, does it *always* fail for that device as long as they do not use desktop mode? - It fails, occasionally it goes trough when using desktop mode or pc. - - Is there anything in common between a lot of the affected devices, as far you can tell? Things like device manufacturer, language, mobile network provider used etc. - Different devices, different android/IOS versions. Only common thing is chrome. PWA is not enabled. - - For one of the devices, can you test if reporting works on WiFi vs. mobile data? - This is something i need to double check. And about my own server, i try to keep running it cheap and mainly training server management. If you need to check hardware specs, I use Raspberry Pi3. All packages should be updated, so PHP7 is installed. Apache+Maria Database (MySQL tables, same engine, so it should not be an issue) Basic firewall configurations for inbound and outbound for traffic. Fail2Ban is installed and configured too. HTTPS is configured. Planning to move that to Azure at somepoint. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Ei viruksia. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> to 2. toukok. 2019 klo 2.25 Marius Lindvall (notifications@github.com) kirjoitti:

…

Hi, and thanks for reporting this issue. I have tested this extensively on v1.1.3 and have been unable to reproduce this with Chrome and Firefox on Android 9, which is the only compatible device I currently have available. The officially supported browsers for FreeField on Android are Chrome and Firefox. I also doubt that this is related to event research as the way it is treated internally is exactly the same as other research tasks. This looks to be a pretty hard to resolve issue, and I will need quite a bit of information in order to troubleshoot it. Could you please tell me the following things about cases when this occurs? - Most importantly, what happens when research submission fails - do you get an error message when you try to submit research, does it submit but not show up on the map afterwards, or does nothing happen at all when you tap the Report button? - Are there any instances at all where reporting this type of research from a mobile browser *does* work (not using desktop mode) when the same task fails on another device? - If reporting fails for one device, does it *always* fail for that device as long as they do not use desktop mode? - Is there anything in common between a lot of the affected devices, as far you can tell? Things like device manufacturer, language, mobile network provider used etc. - For one of the devices, can you test if reporting works on WiFi vs. mobile data? Also, since you mentioned you host your own server: - Are there any PHP notices/warnings/errors/fatal errors in your server's error logs that appear around the time users try to submit research? - Is the server running PHP 7, or the older, outdated PHP 5? - Are you using another HTTP and SQL daemon combination than Apache + MySQL? - Are you using a web application firewall (WAF), load balancer or reverse proxy (CloudFlare-type) in front of your server, or have you configured your server to block some connections based on the User-Agent or IP address of connecting clients? - Do you use HTTPS? I might have to ask for some additional information later, but for now, whatever you can provide of what I asked above will be very helpful in trying to figure out exactly what is going on here. And lastly, sorry for asking this, it might seem like a stupid question but I need to know this for verifying the issue - when reporting research that offers an "[n] prize" reward, can you confirm that the additional box labeled "Quantity" underneath the selected reward is correctly filled with a value and not left blank when clicking Report? Thanks for the report and information so far. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALN7B7JHCE3H3GITZQ5TJY3PTIRFHANCNFSM4HJU3VOQ> .

[bilde2910]

It doesn't seem like the video attachment got through with your email. Could you try submitting it on this issue from the GitHub website instead? It also sounds like something on your server is blocking connections based on the browsers' user agents and/or HTTP request method. I don't know what kind of fail2ban or mod_security configurations you're running, but if this is the only filtering you're using in addition to a basic firewall, I'm somewhat inclined to believe the issue is there, but I can't tell for certain.

Could you try the following in order and see if this has any effect?

• Do you use mod_security in Apache? If so, can you please ensure that the HTTP request methods PATCH, PUT and DELETE are permitted on the same level as GET and POST? Some configurations (see this question for example) may block or rate-limit these types of requests by default. FreeField uses these methods because research submissions are done against a REST API built into FreeField. If one of your fail2ban jails uses a filter regex against GET or POST, ensure that PATCH, PUT and DELETE are also included in that regex.
• In the FreeField admin settings pages, go to the Security page and set "User-agent validation" to Disabled. Save, then on the phone, sign out of FreeField (if you're signed in), then sign back in again and try to report research. If you're not signed in, you can try clearing browser cookies after changing the setting.
• Also in the FreeField settings, go to Permissions and ensure that the "Report field research" permission is set to the same, or a lower value than "Default user group" at the top of the same page.
• Lastly, go to Site settings in FreeField and double-check that "Installation URI" starts with "https".
• Whitelist the IP address that your phone is connecting from in fail2ban and see if this has any effect. Check that the IP address is not banned by fail2ban.
• Temporarily disable fail2ban and mod_security (if using it) and see if submission works when these modules are turned off. If they do, then it would indicate that the error is somewhere in the configuration of these modules.

When making changes to FreeField configuration, make sure you do a simple refresh of the page on the phone before you try reporting research again.

Your environment otherwise looks good; Apache with PHP 7 and MySQL provided by MariaDB is well supported and is what I also run for a few FreeField installations without issue. Firewalls should not be an issue.

If any of this, or none of it works, let me know and I'll see what I can do as the next step.

[Foxpire]

Hi. I did some checks and there is no mod_security installed in Apache. Fail2ban is configured to make ssh more secure, ban if try to bruteforce. - In the FreeField admin settings pages, go to the Security page and set "User-agent validation" to Disabled. Save, then on the phone, sign out of FreeField (if you're signed in), then sign back in again and try to report research. If you're not signed in, you can try clearing browser cookies after changing the setting. This seems to fix the issue. Those rest are actually checked to be ok. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Ei viruksia. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> to 2. toukok. 2019 klo 12.38 Marius Lindvall (notifications@github.com) kirjoitti:

…

It doesn't seem like the video attachment got through with your email. Could you try submitting it on this issue from the GitHub website instead? It also sounds like something on your server is blocking connections based on the browsers' user agents and/or HTTP request method. I don't know what kind of fail2ban or mod_security configurations you're running, but if this is the only filtering you're using in addition to a basic firewall, I'm somewhat inclined to believe the issue is there, but I can't tell for certain. Could you try the following in order and see if this has any effect? - Do you use mod_security in Apache? If so, can you please ensure that the HTTP request methods PATCH, PUT and DELETE are permitted on the same level as GET and POST? Some configurations (see this question for example <https://stackoverflow.com/q/48810247>) may block or rate-limit these types of requests by default. FreeField uses these methods because research submissions are done against a REST API built into FreeField. If one of your fail2ban jails uses a filter regex against GET or POST, ensure that PATCH, PUT and DELETE are also included in that regex. - In the FreeField admin settings pages, go to the Security page and set "User-agent validation" to Disabled. Save, then on the phone, sign out of FreeField (if you're signed in), then sign back in again and try to report research. If you're not signed in, you can try clearing browser cookies after changing the setting. - Also in the FreeField settings, go to Permissions and ensure that the "Report field research" permission is set to the same, or a lower value than "Default user group" at the top of the same page. - Lastly, go to Site settings in FreeField and double-check that "Installation URI" starts with "https". - Whitelist the IP address that your phone is connecting from in fail2ban and see if this has any effect. Check that the IP address is not banned by fail2ban. - Temporarily disable fail2ban and mod_security (if using it) and see if submission works when these modules are turned off. If they do, then it would indicate that the error is somewhere in the configuration of these modules. When making changes to FreeField configuration, make sure you do a simple refresh of the page on the phone before you try reporting research again. Your environment otherwise looks good; Apache with PHP 7 and MySQL provided by MariaDB is well supported and is what I also run for a few FreeField installations without issue. Firewalls should not be an issue. If any of this, or none of it works, let me know and I'll see what I can do as the next step. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALN7B7K2YXLTH7EWIY7JX3TPTKZCJANCNFSM4HJU3VOQ> .

[bilde2910]

Hi,

If disabling user-agent validation resolves the issue, I'd recommend keeping it disabled as the resolution to this issue, as the security issue it protects against is an edge case.

I'll see if I can find the root cause of the issue. Is it possible for you to do a research report from your phone, and then send me the relevant lines from the report from your Apache access log, showing the user agent? You can remove the URL, IP addresses, etc. from the log. I only need the user agents for this purpose. This will help me do local testing to see if there are any notable differences. I would be very grateful if you could provide this :-)

Otherwise, I will add a note in the FAQ and make note of this resolution should it ever come up again. It's a really strange issue and I'm not able to reproduce it locally.

[Foxpire]

Hi, here some info from access logs (access.log) [08/May/2019:19:33:54 +0300] "GET /api/poi.php?updatedSince=-15 HTTP/1.1" 200 3565 "/" "Mozilla/5.0 (Android 8.0.0; Mobile; rv:66.0) Gecko/66.0 Firefox/66.0" Mapaccess.log: ::1 - - [08/May/2019:19:31:16 +0300] "OPTIONS * HTTP/1.0" 200 110 "-" "Apache/2 (server) OpenSSL/1.0.2r (internal dummy connection)" Error logs are empty after this Monday, due the adjusting settings. I'm not software developer, but it seems that there is some issues how server sees the browsers (mobile vs desktop). That could also be at the server side, depending how fast server can reply. Making few adjustment that problem is more random and rare, and it seems only happen when using mobile browser. Though this could also be user side issue (multiple active connections, not up to date browser and so on). I also made debug stop, in case of the need to check if some quests are not reportable. My server settings may not be perfect, but some fine tuning and more practice with this. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Ei viruksia. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> ke 8. toukok. 2019 klo 11.32 Marius Lindvall (notifications@github.com) kirjoitti:

…

Hi, If disabling user-agent validation resolves the issue, I'd recommend keeping it disabled as the resolution to this issue, as the security issue it protects against is an edge case. I'll see if I can find the root cause of the issue. Is it possible for you to do a research report from your phone, and then send me the relevant lines from the report from your Apache access log, showing the user agent? You can remove the URL, IP addresses, etc. from the log. I only need the user agents for this purpose. This will help me do local testing to see if there are any notable differences. I would be very grateful if you could provide this :-) Otherwise, I will add a note in the FAQ and make note of this resolution should it ever come up again. It's a really strange issue and I'm not able to reproduce it locally. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALN7B7I4NP3HG4OJDSMZY3TPUKFYTANCNFSM4HJU3VOQ> .