Skip to content

compare doesn't work with scientist version higher than 1.6.0 #67

Open
@viamin

Description

@viamin

I have written an experiment to compare the output of 2 rails serializers (lightly edited from the production code, but the logic is identical):

def index
    LabTech.science 'PagesController#index' do |experiment|
      experiment.use do
        @pages = Page.all
        @json = ActiveModelSerializers::SerializableResource.new(@pages, each_serializer: PageSerializer, root: 'pages', include: '**').as_json
      end
      experiment.try do
        @json = { pages: QueryService.all_pages }.as_json
      end

      experiment.context params: page_params.to_h
      experiment.compare do |control, candidate|
        control_8601 = control[:pages].map { |hash| hash.deep_transform_values! { |value| value.is_a?(ActiveSupport::TimeWithZone) ? value.iso8601(3).to_s : value } }
        # binding.pry
        control_8601.to_json == candidate['pages'].to_json
      end
      experiment.clean { |value| value.with_indifferent_access[:pages] }
    end
    render json: @json
  end

When I uncomment the binding.pry and check control_8601.to_json == candidate['pages'].to_json, I get true. But when I get the summarized_results, they are always marked as mismatched (this is in my dev environment):

> LabTech.summarize_results 'PagesController#index'

--------------------------------------------------------------------------------
Experiment: PagesController#index
--------------------------------------------------------------------------------
Earliest results: 2023-07-21T22:03:56Z
Latest result:    2023-08-09T05:19:14Z (18 days)

24 of 24 (100.00%) mismatched
--------------------------------------------------------------------------------

When I check LabTech.compare_mismatches 'PagesController#index', there is no difference between the control and candidate.

I'm running the comparison with the deep_transform_values because in my control, the created_at value is coming back as an ActiveSupport::TimeWithZone and in the candidate it is returned as a string - that's the only difference between the two that I can find, however they seem to be identical after the transform. I has noticed this before doing the transform, compare_mismatches would still come back without any difference, so my theory is that once serialized to Postgres and deserialized later, the dates would be normalized.

My Gemfile.lock has the latest versions of the gems:

lab_tech (0.1.9)
scientist (1.6.4)

What am I doing wrong here? Or is there a bug? Could this be related to #41?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions