- Web Scraping Tools Open Source
- Free Web Scraping Tools
- Free Online Web Scraping Tool
- Best Web Scraping Tools
Click here for the latest findU news!
findU is a database archiving weather, position, telemetry, and message data. The primary source ofdata is an amateur radio system called APRS, some weather datacomes from an internet based system called the Citizen Weather Observer Program. This large (58 GB) database is hosted on two servers using data replication techniques. the data is constantly updated (about 20 new reports come in every second),and is accessed via a number of dynamic web pages. Some examples:
Reports are also archived from an amateur radio payload aboard the International Space Station which relaysAPRS reports from amateur radio operators around the globe.
Click any web data to extract - Octoparse applies advanced machine learning algorithm to accurately locate the data at the moment you click on it. Open a website in the built-in browser and start a scraping task by clicking and dragging. Octoparse handles all the messy work behind the screen for you.
Among findU's most significant uses is the forwarding of over a hundred thousand near-realtime weather observations to the National Oceanographic and Atmospheric Administration daily, where the data is checked for accuracy, and used by NOAA and the National Weather Service in a variety of ways.
- Online Services About Services Search Our Records and Documents Today's New Civil Filings Today's New Criminal Filings Forms Docket Search eSubpoena Basket Purchase Copies Public Information Act Requests Public Reports Public Datasets.
- ScrapingBee review. I know I know It sounds a bit pushy to immediately talk about our service but.
Web Scraping Tools Open Source
Here is a page that describes all of the CGIs which are available on this server.This sometimes lags behind the latest developments, but it is the best place to start. Checkthe news link at the top of this page, and especially the APRS SIG at TAPR, forthe very latest news.
FindU is a database archiving weather, position, telemetry, and message data. The primary source of data is an amateur radio system called APRS, some weather data comes from an internet based system called the Citizen Weather Observer Program. ParseHub is a free web scraping tool. Turn any site into a spreadsheet or API. As easy as clicking on the data you want to extract.
Directly creating a URL by editing the parameters is not meant for beginners, though by readingthe documentation and experiementing a bit, anyone should be able to get the results they areafter.Other people have written web pages that allow you to enter requests to findU via forms, this is an easier way to build the URL you need. I'm sure I've missed some, email me your favorites.
Free Web Scraping Tools
| URL | Description |
| http://www.wulfden.org/APRSQuery.shtml | N1BQ's page, the original |
| http://www.qsl.net/n9wtm/ | N9WTM |
| http://www.phrozen-neon.com/alarc/aprs.php | N0ESC |
| http://www.proaviator.com/findu/ | K4IFR |
email questions and comments to Steve Dimse steve@dimse.com. First, make sure the question isn't answered in the FAQ below...
findU FAQs
Can I track my car (or anything else) on findU?
Absolutely. The system is open to all amateur radio operators. A license is required, but it is not difficultto obtain. Ifyou are in the US, and want information on obtaining a license, see the ARRL web site for info. Also, there are many commercial sitesthat can do tracking without using amateur radio.I have an amateur radio license, where can I find out how to get started in APRS (or any other APRS question not concerning findU)?
Free Online Web Scraping Tool
I'm sorry, but I'm not able to answer these queries due to a lack of time.At one time I kept a few links to beginner web sites here, but they come andgo too fast for me to keep up. Use Google to findsome that are available now. TAPR's aprssig is a good place to ask these questions,you can sign up here.Why does my station's information not show up on findU?
If you are a member of the Citizen Weather program, the most common problem is an incorrectlyentered latitude and longitude. Some programs do not check the number enetered for validity, butthe system has a very specific format it requires, so check that the information is entered correctly.For other problems, you should get support from the person or company that produces the programyou are using.The process of reports getting to findU via amateur radio and APRS is complex. First, your station transmitsthe data, likely it must travel through one or more digipeaters before reachingan internet gateway, which forwards it to a hub in the APRS Internet System,and then finally to findU.com.
Problems can occur anywhere along this path, though by far the most common are in the early stages...an individual's RF system has a problem, their path is not set correctly to be digipeated by their local RF network, or their local IGate is down or misconfigured. From the internet there is absolutely no way I can diagnose these problems. Your best bet is to seek out a local APRS user that can listen for you on RF, and try to figure out what is wrong. If you are writting your own software, or manually entered an APRS string into a hardware or software configuration file, you may have misformatted the data. findU tends to follow strict interpretation of the APRS standard, and some errors may be displayable in some client programs but not findU, a typical example is an incorrect number of digits in a lat/lon. You can always check findU's parsing error page to see if findU is detecting an error.
On the weather page, it says there is no radar within 150 miles, but there is!
Best Web Scraping Tools
Currently findU only has US radar sites, if you are in the US and this is happeningto you, let me know.I have no problems with adding other radars, but the following requirements must be met:
1. The radar image alone must be available in GIF or PNG format with a fixed URL and constant scale. I may be able to work with JPEG or other graphic formats, if everything else is possible let me know.
2. Image provided without any copyright restrictions (this is the big gotcha,I've not found any outside the US that meet this requirement). Here is the USNational Weather Services statement: 'The information on government servers are in thepublic domain, unless specifically annotated otherwise, and may be used freelyby the public.' If you are submitting another site, either submit the URL of a page stating there are no restrictions, or the contact information of theperson that stated the use was acceptible.Some countries allow non-commercial use, and while findU itself is non-commercial, I am unable to guarantee how people use findU, infact I know of several sites that do use findU for commercial purposes. Commercial use must be acceptible, or an exception made for findU, I will notaccept the risk of copyright violation.3. Someone must create a geo file for the image(s). This is detailed on http://www.findu.com/geo.htmlI do not have time to do all this, so if you want your local radar images, itis up to you.Why don't you track digipeaters, so I can see who is using my digi (or any other RF propagation related issue)
This is a matter of findU (or more generally the APRS Internet System) beingthe wrong tool for the job. My design of the APRS IS was based on getting datadistributed widely, not about characterizing the RF network. Therefore, the systemis designed at each stage to filter out duplicate versions of the same datapacket. However, it is precisely the differences between different instancesof the same packet that would allow you to draw interesting and/or definitiveconclusions about the RF network and propagation.The decision in favor of duplicate filtering was, and still is, the correct one...without it, the APRS IS data stream wouldbe perhaps ten times its present size, which is already big enough to cause problems. PropNet nowhas set up an alternate network which is suitable for propagation studies.
Is the source code available?
No. Commercialization is the primary reason I've not open-sourced my code. I've gotten a surprising number of requests from persons inside the GIS industry to either give or sell them my code, or consult for them. After looking at a lot of the commercial systems, the entire APRS system really holds its own, and on many counts including (in-my-not-so-humble-opinion) findU, it is better than anything else out there. Winning the Earthlink AVL competition to me was objective confirmation of this opinion. If my code were GPL'ed, it would probably be used in commercial systems. I get my satisfaction from producing a useful system that does cool things, and letting people use it for free.I had expected that the Linux Lunacy Geek Cruise I went on October 2001 would convince me once and for all of the benefits of free (as in speech) software. In fact, it had the opposite effect...I spent a lot of time with the infamous RMS (Richard Stallman, president of the Free Software Foundation) since we ate dinner at the same eight person table. Let's just say the discussion was as enjoyable as, and far spicier than, the Holland America food! After all the arguments and much contemplation, I've come to agree with the equally infamous ESR (Eric Raymond, also on the cruise, talk about storm-at-sea!), that RMS and FSF are seeking power (defined as the ability to make decisions that affect others), not freedom (the ability to make decisions that affect yourself). Now that I really understand it, I find the FSF vision of the world a scarier place than even a Microsoft controlled world. Software is a creative work (RMS agrees), the disposition of which rightfully belongs in the hands of the creator or their employer (RMS turns red and starts screaming, because useful creations like a home improvement TV show, how-to book, or software program, morally belong to the collective). I believe the decision to donate code for the public good is an altruistic act, not a moral imperative, and a choice deserving admiration, not expectation.
There are other reasons besides commercialization, like the amount of time it would take to document,distribute, support, and re-integrate code from an open source project.If you want the code, think about the reason for a minute. Is it becauseyou want to see the magic? Nothing magic, this is a simple, though large, databasedriven web application, there are hundreds of examples you can view on the web, and dozens of books thatyou could read on this topic. It is to create your own findU site? How useful would that be, and if you dodraw users away from the original, how can the public be sure any bugs or improvements will be on your site?Perhap to change one thing that bothers you and send it back to me? When am I to find time to judge the worthiness of all the modifications? Something else? I am always willing to listen to other viewpoints,but I need a compelling reason to open the code, and I just haven't found it yet.
My choice for now is to keep control of the code, and provide a service to the community for free (as in beer!). Not a purely altruistic decision, but the one I am most comfortable with. In the event I someday lose interest in findU, it will be passed to someone else for development. Over the years many have offered donations,and I've decided to accept them now, as well as place Google adds on the pages. None of this willmake me rich, but perhaps snough will accumulate to eventually pay for the next server findUwill run upon.
Why don't you add...
If I never came up with another idea, the things I alreadyhave on my list will keep me busy for a couple of years. In other words, I havelots of plans that I haven't implemented, so I may already to be planning todo it. However, if you have an idea you'd like to share, please do so, for tworeasons. First, it may be something greatI hadn't thought of, and second, my priorities are to some extent dictated bywhat people want to see, so voting for what you'd like may get it done sooner.However, please don't be hurt or surprised if I don't answer your email quickler(or sometimes at all). I do my best, but at some point it becomes counter-productive...I'drather code than write email.Can I publish links to the dynamic pages, or incorporate the dynamic images in my own page?
In general, yes. Keep in mind this system is in heavy development. The CGI interfaces to any of the pages and images may change without prior notice. If you use these links, then it is your responsibility to check them often to assure that they still function! Please don'temail with broken link problems!!! Dynamic findU HTML pages must not be used to extract data which goes into a database...in other words, no screen scrapers. If you have a project that needsto do this, you should either use another database as a backendor create your own database by parsing the APRS data stream. Several open source APRS parsers are available.Lately there has been a great increase in the number of people attempting screen-scraping, so I mustclarify my position. A single access generated from a user action (starting a program, pushing a button, etc.)is allowed, while any repetetive access by a program is NOT ALLOWED. This is necessary because a single program can use as much resource as dozens of human users, and I do not have the capacity to support thissort of use. I will be monitoring for inappropriate access and blocking IPs where I find a problem.Recent events force me again to clarify this. If you use findU upon some user-generated action, please think hard about the most efficient way to accomplish this. I've had peopleuse findU to show unaltered radar images by using an off-the-radar station. findU had to fetch the image, realize there was nothing to draw, then send it out again. All this instead ofthe person getting the image directly from the NWS. Others grab a list of hundreds of raw reportsand use one. Please act as if you realize that findU is the limited resource it actually is.If you have questions about any use please ask.
Can I use findU for special events?
Yes! There are a bunch of examples, from the Big Ride (3000 mile, 6 week biketrek across the US) to balloon races, marathons, road races. The easiest way (from myperspective anyway) is for you to write your own web pages and insert findU imageslike plot, breadcrumb, and radar-find using IMG tags in your html. There areother things I can do, like writing a custom web page, as I did for the largest APRS event, the Baker to Vegas run or theSuitSat experiment on the International Space Station. This sort of thingtakes time, so I need to limit it to things that will get some main-stream publicity(I love to show off what ham radio can do). Please, if you want this, contactme as soon as possible, as my schedule sometimes gets tight! The more lead time you give me,the more likely I am to produce something really cool for you.Your images don't show on my browser!
This site uses PNG images (Portable Network Graphics) for the dynamic images rather than GIFs. This was because UniSys held a patenton the compression used in GIFs, PNG is an improved format without patent problems. All of the latest browsers accept these graphics, and ones that are a bit oldercan handle them through Apple's Quicktime plugin. Recently these patents have expired.The library I use to create theseimages, GD, is now available in a form that will create GIF images. Had this been the case when I firstcreated findU, I certainly would have used GIF. At this point, PNG is close to a universalformat, and I do not feel converting my code to GIF would be worth my effort. If you have a differentopinion, please feel free to share your reasons with me.I'm with SuperMegaWeatherSite.com, I want to use your data!
First, this is not my data, and I do not feel I have the right to allow other people's datato be used for commercial purposes.Technically the data is not copyrighted, it is legal to write a your own APRS parser to extract thedata from the live APRS internet stream, but because commercial use of findU would increase the load without giving anything back to the APRS community,I must insist you not use my server to harvest the data.About the findU server