What are some of the best web data scraping tools?

What are the best tools (free/paid) to scrape large amounts of data from webpages and store it in required formats?  The tool preferably should be automated to look for new data at specific intervals. Answer here.

Do you JSON?

SQL Server 2016 ♥s JSON! JSON is everywhere – it’s popular textual data format used for exchanging data in modern web and mobile apps and storing unstructured data in log files or NoSQL databases like DocumentDB. SQL Server 2016 provides built-in function and operators for JSON.

Get historical results without being blacklisted for scraping.

Where do you get historical results without being blacklisted for scraping.

For historical traders particularly, here are Betfair sites Betfair:Results Feed and Betfair Data Home    to source historical data with out resorting to scraping.  Ideally we should be able to request results from Betfair API-NG with parameters, so we can filter the data results we need and already in JSON format which can be easily displayed or update your database, if you have one. "Food for thought".

JSON sounds foreign to me.

JSON sounds foreign to me.

How I skin this cat JSON using classes (objects in object oriented term). Before you can send JSON rpc call (this is the protocol used by JSON as different from SOAP in the  old API 6.0) you have to  create classes for the request where Json picks the values for your params and filters assembling (serialization) and then send the request to Berfair servers. The server will send back the response and Json will fill the response(deserialization) classes for you to fill your datagridview. You can not fill your datagridview directly, you have to bind the response to a dataset and datatable and then fill the datagridview from the response classes.  How do you know what classes to create ? for request use the Betfair Visualiser select any of the listed requests and cut a paste the value returned to Json Parser and study the tree like hierarchy. From there you can create response classes that you need. For traders learning how to program VB Json rpc method is understandable and easier to follow. See it's not that hard. That's how I' skinned this cat Json.

VB online reference for programming your Betfair API-NG application.

Visual Basic is the most productive tool for rapidly creating a wide range of Windows, Web, Mobile, and Office applications built on the .NET Framework. The Visual Basic language is designed to be human readable and accessible to everyone from novice programmers to advanced programmers. 

Learn more about VB here or here. E.g. DataSet, DataTable, DataView, Datagrid and etc.

Betfair UK and AU now share a single wallet and API end point.

Important Betfair update now finished.

Betfair has now ONE WALLET after removal of Australian wallet and single API end point for UK and AU customers. The Betfair Exchange migration is now complete and successfully test my app using this API end point: "https://api.betfair.com/exchange/betting/json-rpc/v1"  Now my app picks all races in UK and AU using one single API end point.

Using APIs Can Add Significant Value.

Vikas Doshi discusses the many advantages of using APIs for data compilation and trading across exchanges, compared to manually trading using traditional user-interfaces.

One of the key factors that contribute towards the growth of the world’s sports trading through exchanges is the API functionality to allow traders to back and lay automatically, rather than manually. It has helped to attract an entirely new breed of player over the past ten years, especially those with a solid technical understanding.

APIs have allowed many of the exchange’s customers to innovate and be creative; some have developed their own customized application over the APIs and others have begun using API products created by other professional developers like Bet Angel, Gruss and so on.

The ability to execute trades automatically is one of the main differentials for groups making the maximum levels of profit from sports trading. So, how does it work? It may seem complicated to the untrained trader, however it is worth the investment of time to become familiar with it. Although an API can be a profitable tool for any serious sports trader, that doesn’t mean developing proprietary software; an off-the-shelf product may be good enough to serve the purpose.

The mainstream exchanges will generally be designed to cater for the casual bettor and ‘punter’ however the professional trader can find that navigation around the interface can be time consuming and irritating for them. For this market, API-based products can be highly beneficial.

So, if you are an independent prop trader or small syndicate; how can you begin to take advantage of such APIs. What can you obtain from the use of an API that you cannot get from a website? The key advantages are; speed, repeatability, accuracy, data management, real time data and automation.

1. Speed & repeatability – different sports traders are looking for different patterns. Using a very simple example might be to say that the ‘model’ should only shortlist those horses running today who are trained by “W P Mullins”. The API can extract this data and it is then easy to get that data on a regular basis.

2. Accuracy – Any retrieval or execution of data using software means there are no chances of human error. Of course all such software has to be thoroughly tested or else it can be a disaster, but it works!

3. Data management – Manual compilation and analysis of data can be very time consuming (and boring), and if you are dealing with ‘big data’ sets then it becomes even more complicated. If that data is retrieved by an API, then one can filter it for only what is required whilst the rest can be discarded.

4. Real time – Exchange APIs do allow you “real time” access; within a few seconds one can scan lots of Exchange Markets OR you can read market prices very quickly. For some traders this is very critical.

5. Automation – This is probably the single most beneficial advantage of using APIs if used intelligently. You can automate your sports trading strategy, take emotions out of it, and it saves time. With the right ‘model’, an API can be almost like a money-printing machine! Of course the key is to have a viable and profitable trading strategy which can be automated or partly automated.
Though accessing APIs might require a large initial investment of time and money, over the long term, mastering this technology can prove highly profitable for the talented sports trader. A wealth of statistical sports data is available on the internet to either be ‘scraped’ or purchased but the winning formula is up to the trader!

Develop Betfair betting application all for free.

Automated Betfair Betting Environment all for free.

If you have a PC or Laptop running Windows 10 Home or PRO. 

You can develop an automatic Betfair betting application using the following:

Integrated Development Environment - Visual Studio 2015 community edition.

Programming Language - Visual Basic part of Visual Studio 2015 community.

Database Management System - SQL Server 2016 Developer Edition with Data Tools.

Data Sources - Betfair API-NG.

Protocol JSON RPC

The DBMS is optional only if you keep historical data to support your betting decisions or you can use it to store data retrieve from Betfair API-NG response.

Using MS SQL Server 2016 developer edition SSIS Data Tools to scrape websites.

Using SQL Server 2016 developer edition SSIS Data Tools (BIDS) to scrape websites for results and prices. The good news is that the above developer edition is now free and can be download here.

Forget the express editions as data tools are not supported on those editions. SSIS project is similar to Visual Studio projects you use to develop windows desktop applications and a lot more. While the projects that you will create in SSIS are for data centric tasks. You can extract data from websites and then use them to update your trading database that supports your betting rules/decisions.

You can query your database outside your trading bot software that will give you more insight on your data. The potential benefit of using SSIS is that you can run it all day long scraping results and prices and automatically updates your decision support database which is independent from your bot software. Your bot software can now access up to date information outside of Betfair API's.

If only Betfair will supply the sports results in their API's, life can even be less difficult.

Betfair betting strategy the weirder the better.

You should design your Betfair bot software betting strategy that it is so unique and weird that you think that no one is using your strategy. You will earn peanuts by scalping, arbitrage, greening and others plus you are always behind the eight ball because you are not using a server near the global exchange to place your orders and that you are so far behind the queue and miss the opportunity of being matched. So design your bot software betting strategy that you can put your bets an hour before start time or even a few hours earlier. So your strategy is not dependent on quick market movements but on the outcome. Of course you can always opt to hedge if the market moves in your favour.

Demystify Kelly Criterion staking formula.

The Martingale and Fibonacci staking strategy depends on previous outcome where you increase your bet until you have a favourable result and restart the sequence again. While the advantage of Kelly Criterion strategy is your bet is independent of your previous bets but on the odds and probability of a single outcome result of any sport game. A simple explanation is a game with odds of 2 to 1 and a probability of 50%; you should not bet as per Kelly criterion. The odds is not the bookmaker odds as they are always smaller because of the bookmaker overround. What your strategy is to design in your bot software to calculate your own odds and probability  as input to Kelly criterion formula to determine the percent of bet you will put based on your arbitrary bank. Pinnacle explains Kelly criterion with calculator.

Bet on Betfair for long competitions.

You will always get a much better odds on Betfair than in other bookmakers for long competitions. e.g.. Golf events, NBA,  Soccer and etc.  You can hedge your bet later on if the market goes your way. You can also program this in your bot software. Betfair hedge calculator here.

Betfair premium charge. It's complicated.

You have to pay extra commissions to Betfair.

When you have:

• Bet on more than 250 markets
• Are in profit over the life of your account
• Have paid less than 20% of your lifetime profit in “total charges”

Explained here by Betfair. So design your bot to cater to keep a history of your bets and try to limit your bets to 250 markets.

Avoiding Betfair''s Premium Charge by Brooke Greville. "Since the introduction of the punitive charge for the most successful players globally, there are some strategies that such players can adopt to minimise their exposure to Betfair’s platform."

Betfair Market Data Request Limits.

Betfair Market Data Request Limits - Be aware on how much data your program requests from Betfair. The base rule is sum(Weight) * number market ids must not exceed 200 points per request. Design your request to work around this limits.

Betting strategies to Bet Better.

Betting Better with Betfair. Can be better accomplished using an app software that you write your own which is secured and can not be monitored by others.

Installing VS2015 community to publish your application.

Install VS2015 community  use "default"  option.  Run the community setup again, select "Modify" and then tick the signing option . This will allow you to "Publish" your application.  Installing this way will prevent the dreaded "SignTool.exe not found" that makes your project unrunnable.