How do I reference the correct cell on a Data Grid View that has been sorted?

How do I reference the correct cell on a Data Grid View that has been sorted?

I have a WinForms application that has some data in a DataGridView. If a
user double clicks on a cell then that row needs to be removed, and that
works fine unless they click on a column first to sort the data by that
column. If it has been sorted then the row index of the event still refers
to the origional value in that row (I.E. Double clicking on the first item
after sorting the data will remove the row that was at the top before the
sort).
Is there a way for me to update the row indexes when the sort occurs?
Current initialize is similar to:
DataTable dt = new DataTable();
NpgsqlDataAdapter da = new NpgsqlDataAdapter(query, connection);
da.Fill(dt);
this.dgv1.DataSource = dt;
And the double click event is similar to:
private void dgv1_CellDoubleClick(object sender, DataGridViewCellEventArgs e)
{
DataTable dt = (DataTable)dgv1.DataSource;
if (e.RowIndex > -1)
{
dt.Rows.RemoveAt(e.RowIndex);
}
}
I suspect the issue is something very simple that I am not seeing right now..