Drawing a hex grid is a matter of formulas, Hex Coordinates shows how. The other direction is the one you need all the time: the mouse is somewhere, which tile does it mean? With a square grid you divide by the edge length and you're done. With hexagons the rows are shifted against each other and the edges are slanted, one division won't get you far.
The card on the left shows the calculation for the mouse position: first the fractional values q, r and s, then the rounded ones, then how far each value moved while rounding, and at the bottom the result. The chip "Naive" rounds the three values on their own, "Rounded" corrects the result. The orange area in the field is where the naive version finds no hex. With "Line" you draw a line between two tiles: a click sets the start, the mouse is the end. Pointy or flat can be switched, the code is the same, only the matrix differs.
The idea comes from the hex tutorial by Red Blob Games, which I recommend a lot. Everything here is in my own words and with my own code.
Pixel to fractional coordinates
Hex to pixel is a linear map, for pointy hexagons it looks like this:
return {l.origin.x + l.size * (kSqrt3 * q + kSqrt3 / 2.0 * r), l.origin.y + l.size * 1.5 * r};
Going back uses the inverse matrix. Subtract the origin and divide by the hex size, then two lines:
const double x = (p.x - l.origin.x) / l.size;
const double y = (p.y - l.origin.y) / l.size;
const double q = l.flat ? 2.0 / 3.0 * x : kSqrt3 / 3.0 * x - 1.0 / 3.0 * y;
const double r = l.flat ? -1.0 / 3.0 * x + kSqrt3 / 3.0 * y : 2.0 / 3.0 * y;
return {q, r, -q - r};
The result has decimals, the mouse sits somewhere between the tile centers. s is derived from q and r so the point stays on the plane q + r + s = 0. All that's left is rounding.
The obvious trap
Three numbers, each to the nearest integer: round(q), round(r), round(s). The catch: every value moves by up to 0.5 while rounding, so the sum of the rounded values is -1, 0 or +1. At -1 and +1 you hold a triple that is not a hex.
You can work out how often that happens. Let a be the distance from q to the nearest integer and b the one from r, both signed. The sum only works out if |a + b| stays below 0.5. In the unit square two triangles in the corners violate that, an eighth of the area each, so a quarter together. In the test I count it over a grid of pixels, over both layouts the share is between 20 and 30 percent. You can see it in the demo: press chip 1 and the orange area runs around the edges of the tiles. In the middle of a tile everything rounds correctly, the failures sit at the borders.
At the start the mouse is close to the corner of a tile: q = 0.554, r = -0.308, s = -0.246. Rounded that becomes (1, 0, 0), sum 1. That is no hex.
The fix
The value that moved the most while rounding has the biggest error. You throw it away and recompute it from the other two:
if (o.dq > o.dr && o.dq > o.ds) {
o.fixed.q = -o.naive.r - o.naive.s;
o.fixedAxis = 0;
} else if (o.dr > o.ds) {
o.fixed.r = -o.naive.q - o.naive.s;
o.fixedAxis = 1;
} else {
o.fixed.s = -o.naive.q - o.naive.r;
o.fixedAxis = 2;
}
In the example q moved by 0.446, r by 0.308, s by 0.246. So q goes and becomes -0 - 0 = 0. The mouse is in tile (0, 0, 0). That really is the tile whose center is closest: in the test I compare 4000 random points in both layouts against a brute-force search over all tile centers, and there is no difference.
If two values are tied, the order of the checks decides. For a point exactly on an edge that is a convention, both tiles are equally close.
A line across the grid
The same rounding carries lines. Between two tiles there are N = distance(a, b) steps, the largest of the three coordinate differences. You take N + 1 evenly spaced points on the straight line through cube space and round each one:
out.push_back(n == 0 ? fa : lerp(fa, fb, static_cast<double>(i) / static_cast<double>(n)));
The result is a chain of neighboring tiles without gaps and without jumps. I test that for many start and end pairs. If you switch to "Naive" in line mode, the sample points whose naive rounding is no hex drop out: orange dot, no tile.
That leaves the edge. If the line runs exactly along the border of two tiles, the tie rule decides. From (0, 0, 0) to (2, -1, -1) the middle point lies exactly at (1, -0.5, -0.5), and without any countermeasure the result is (1, -1, 0). The chip "Nudge" shifts the whole line by 0.000001 (precisely by (e, e, -2e), the sum stays zero):
const Frac fa{a.q + e, a.r + e, a.s - 2.0 * e};
const Frac fb{b.q + e, b.r + e, b.s - 2.0 * e};
Now the point is clearly on one side, the result is (1, 0, -1), and it doesn't matter in which direction you walk the line. The test has exactly this case. In the demo you can see it if you set the start to the middle tile, put the end two tiles to the upper right of it and press N: the tile in the middle of the line jumps to the other side.
Pitfalls
The conversion to int is where mouse coordinates outside the window or a NaN can go wrong. toInt clamps to plus or minus a million and turns NaN into zero, so the cast never causes undefined behavior. A hex size of 0 would divide by zero in toFractional, there the origin tile comes out.
And the orientation: pointy and flat need different matrices and different corner angles (a 30 degree offset). If you mix formulas from pointy and flat, you get tiles that only almost meet. So the test checks that two neighboring tiles share exactly two corners in both layouts.
The code is in the repo under hex-pixel-runden. The project Game Dev Snippets collects all demos of the series.