I was unable to get the matrix:total_rows tag to work with the front-end data fallback.  I believe the issue is at line 122 in Publisher_matrix_hooks.php.  Matrix is using the SQL count function in the total_rows query so it will always return a row even when there are none in the current language.

I replaced line 122:

// If no rows were found, and fallback is set, then query default language instead.  
        if (ee()->publisher_setting->show_fallback() && $query->num_rows() == 0)

with a check of the actual value being returned vs the num_rows and it is working so far:

// If $select_mode == 'aggregate', then the rows returned will always be > 1, check value instead
        if( $select_mode == 'aggregate' )
        {
          $result = $query->row_array();
          $agg_count = $result['aggregate'];
        }


        // If no rows were found, and fallback is set, then query default language instead.  
        if (ee()->publisher_setting->show_fallback() && $query->num_rows() == 0 || $select_mode == 'aggregate' && $agg_count == 0)