Data Operations Engineer

Job description

The Role

We are looking for a talented Data Operations Engineer to join our team. We need proactive and passionate team players with a drive to learn, and to improve, our tech and processes. You'll be integral to the handling of industrial quantities of complex data as we have over 200 data feeds to keep running smoothly. We have also started an ambitious mission to rapidly expand the freshness, breadth and depth of the data we collect.

Our Data Challenges

We need to focus on efficiency and productivity improvements in order to maintain our competitive advantage, scale at pace, provide better and fresher data and minimise human interaction:

  • To do this, we intend to build a new data pipeline - allowing us to structure, systematise and automate our activities from data sourcing, fetching and ingesting of data through to provision to clients.

  • We need to be able to handle industrial quantities of complex data as we have hundreds of data feeds to keep running smoothly, and we have started an ambitious mission to rapidly expand the breadth and depth of the data we collect.

  • We need to accelerate our data acquisition and remediation, analysis and on-boarding processes and technology whilst maintaining a high level of accuracy and documentation

What you'll be doing

  • Overall you will manage our BAU data pipeline (our systems that fetch and ingest incoming data into OpenCorporates) and ensure the smooth running of our data operations (including ensuring the timely and accurate delivery of data to our clients)

  • You will monitor the data environments (e.g. through our dashboards and logs) to identify and escalate issues

  • You'll be diagnosing and resolving issues that occur, working as needed with other team members in our Data and Tech teams

  • Improving the way we work. You will be a key player in our goal to have a ruthless focus on efficiency and productivity improvements in order to maintain our competitive advantage, scale at pace, provide better and fresher data and minimise the human interaction
    • Participating in the design of new processes and dashboards to efficiently manage our BAU tasks and allow us to see at a glance the current status of our 200+ data feeds

    • Working with the team to structure, systemise, automate and document our data pipeline to ensure our systems are scalable and reliable

  • Making sure that our internal stakeholders receive quality and timely status updates, issues are escalated promptly and queries are answered

  • You will work with colleagues to ensure that clients and users receive timely answers to questions about our data and you will play a key part in documenting these answers to ensure that our clients and users have the answers at their fingertips

  • Weekend and/or out of hours work will be required, including the monitoring of exports and bots

Requirements

Relevant Technical Skills

3 + years experience with data in a data analyst, data engineering or master data management roles. Above all we are looking for talented people who we think will fit in well, who are happy to be part of a team - sharing knowledge and learning from each other.

Desirable Skills

  • Query and understand structured data within SQLite, MySQL and JSON

  • Software development knowledge in Ruby or Python, Git, Linux shell scripting & tools

  • ETL processes and data pipelines; data testing/quality assurance processes (script based languages as opposed to ETL tools)

  • Data analysis and quality management on large datasets

  • Root cause analysis & data remediation experience

  • Accuracy and attention to detail.

  • Excellent verbal and written communication skills

Useful but not a prerequisite:

  • Process improvement and automation

  • Web scraping

  • Elasticsearch

Our Values

Our values outline the shared principles that define the OpenCorporates culture and team environment. The company values underpin everything we do, whether that's through who we chose to work with (and who we chose not to) to day-to-day decision making, teamwork, supporting our clients and evaluating individual and company performance, the core values are the lens we look through in everything we do.  All our employees are driven by our values and use them as a compass to guide their work and collaboration with colleagues and clients.

Be Bold & Beat The Odds

  • Our work is hard - and matters. We will succeed by being more ambitious, more imaginative and more daring than our competitors

We Put Users First 

  • Success will only come if we focus obsessively on the success of our users in everything we do

Learn & Adapt

  • There is no straight line to success. We will excel by taking a scientific approach to all our work

One Team

  • We win together. We fail together. And diversity – of backgrounds, of views, of personalities – is a critical asset