Well Placement for Drilling Using Reinforcement Learning

Maria Ulyanova25 May 2024

Problem

An oil company places new wells on a field. An expert studies the 2D characteristic maps of the hydrodynamic model, picks rough areas, runs a simulator that returns production, and repeats until all wells are placed. The loop is slow, and the result depends on where the expert looked and on empirical habits. A better layout yields more accumulated production, and a bad one is paid for over decades.

Example from the article: on a real field the agent's layout gave +3.7% accumulated production up to 2050 against the expert's layout (case 2).

Solution in brief

Treat placement as a reinforcement learning (RL) game on a 256×256 map. The agent picks one cell per step, and a simulator-based reward calculator returns accumulated production. A convolutional Deep Q-Network (DQN) outputs a value map over all cells, and the best cell is the next well. At inference no simulator run is needed.

Questions this case answers
  • How to choose locations for a set of new resources on a spatial map by sequential decisions, using a slow physics simulator only for the reward and a learned value map at inference
  • How to replace a slow expert simulate-and-adjust loop with a learned sequential planner whose output is compared with the expert plan by a cumulative simulated outcome
  • How to optimize drilling well placement with reinforcement learning against a reservoir simulator
  • How to choose coordinates on a 2D map with a DQN that outputs a value for every cell

Tools and methods

  • Ape-X DQN with prioritized experience replay — trains the agent from a shared buffer of transitions.
  • Convolutional encoder with value and advantage heads — outputs a 256×256 value map.
  • Reward calculator over the hydrodynamic model — returns accumulated production for a layout.
Continue reading — it's free

Google or a one-time email code. No password.