Data-gather Project

Data project to gather all available job lists on CvKeskus.

Data gatherer with mouse over function

This project was basically introduction for data engineer because i wanted to learn python and capabilities of it and you know, i love to handle big data so at first i went to CVkeskus website and checked how the HTML syntax is built on website and programmed python where those data fields are set and managed to pull off some data. at first i managed to build only on first page but usually on website, you have multiple page and when i scraped data, it totally had around 1100 free jobs so basically on every page was around 50 jobs. It had lots of pages and had to figure out with python how to fully automate on next page and scrape more data and also not mess up CVKeskus website it self. I built on python it scrapes slowly to it doesnt disturb website network.

When i managed to get first stage of data gathered and prepped up my mysql database with tables, had a interesting thought hmm.. how about map out also on where these jobs are with coordinates because some of these jobs are like at another city.

Data gatherer with mouse over function

Data gatherer with mouse over function

For me it was even more interesting task and started to love even more data engineer field and when i managed to get coordinates.

Here's little bit more zoomed inside Capital of Tallinn

These blue pins are companies location.

This is zoomed inside Tartu.

Coordinations of companies

success rate of get coordination was basically around 75% because if company had more than 1 building then python automation couldn't detect which buildings to get coordinations it failed. It could be improved but at that time around i thought that was enough for me.

most of the available jobs wasn't salary available.

here it's possible to see some even post salary which is good.