Measurementinsocialsciencesdivergesfromthestraightforwardquantificationofphysicalattributessuchasheightorweightbecausethemeasuredtraitsarelatent,existingbeyonddirectobservation.Thesemeasurementsconsistofaconsiderableamountoferror,whichneedstobeaccountedfor,andmultipleratersand/ormulti-iteminstrumentsareinvolved.Consequently,aspectrumofstatisticalandpsychometricmodelsandtechniqueshavebeendevelopedtoanalyzesuchmeasurementsinthesocialsciences\citeprao2007psychometrics,bartholomew2008analysis,martinkova2023computational.Thesemethodologiesincludeprovidingproofsofthemeasurementreliability,andassessingvaliditybyanalyzingrelationshipswithcriterionvariablesandbyanalyzingtheinternalstructurewithfactoranalysis.Giventhatsocialsciencemeasurementtypicallyentailsmultiplecomponentssuchasitems(orcriteria,raters,occasions,etc.),thereisaparticularfocusonmodelingitemresponseswithinmulti-itemmeasurementsandcheckingthefunctioningofeachindividualitem.
Asthefieldanddatacomplexityundergoarapidevolution,itisimperativetoenableuserstoextendtheanalysesandfunctionalitiesoftheSIAapplication.Toachievethis,weareintroducingthe"SIAmodules"feature.Thiscapabilityempowersresearchersandpractitionerstodevelopadd-onSIAmodulesthatseamlesslyintegratewithandexpanduponthecapabilitiesofthemainapplication.Indoingso,wetakeinspirationfromjamovi\citepjamovi,JASP\citepjasp,andRpackagesRcmdr\citeprcmdr,Deducer\citepdeducer,andRKWard\citeprkward,allofwhichofferextensionframeworks,thusbeingsimilartoourendeavor.Apartfromtheaforementionedsoftware,boththeSIAapplicationandSIAmodulesarewrittenentirelyinR,keepingitopenforthewiderRcommunity.
ThemainSIAapplicationincludescorepsychometricmodelsandmethodsforevaluatingmulti-itemmeasurementinsocialsciences,suchasprovidingproofsofmeasurementvalidity,modelstoassessreliability,andproperfunctioningofeverysingleitem.Moreover,userscanintegrateadditionalanalysesandadvancedapproaches,suchascomputerizedadaptivetesting,textanalysis,andothers,throughtheSIAmodules.
Theconceptofvaliditydescribesthedegreetowhichanassessmentinstrumentmeasuresitsintendedconstruct.Quantitativeevidencefromvarioussourcescanbeemployed,withvariousstatisticalmodelsavailabletodemonstrateandassessmeasurementvalidityfromdifferentperspectives.Onesuchsourceofevidenceisprovidedbyanexternalvariable(criterion)measuringthesameconstructandemployingcorrelationcoefficients,ttitalic_ttests,analysisofvariance,orregressionmodels,dependingonthedatastructure.Additionally,theinternalstructureofthetest,whichisanimportantaspectofvalidity,canbeexaminedthroughfactoranalysis.
Reliability,ontheotherhand,referstotheconsistencyofmeasurementsandtheextenttowhichtheyareaffectedbyerror.Intherealmofmulti-itemmeasurements,onewayofassessingreliabilityisthroughtheanalysisofinternalconsistency,whichinvolvescorrelationsbetweenitemsorbetweensubscoresobtainedfromsplitsofthetest.Anotherapproachentailsanalyzingcorrelationsbetweenscoresderivedfrommultipletestadministrationsorassessmentsbymultipleraters,whereapplicable.Assessingreliabilityinmorecomplexdatastructuresmaynecessitatevariancecomponentmodels,offeringaflexiblealternative\citepmartinkova2023assessingIRR.
Analyzingitemresponsesisimportantfordevelopingmulti-itemmeasurementsandforadeeperunderstandingoftherespondent’straitsandcomponentsofthemeasuredconstruct.
Forinstance,considermodelingtheprobabilityofacorrectansweroritemiiitalic_iendorsementonaperson’sppitalic_pabilityθpsubscript\theta_{p}italic_θstart_POSTSUBSCRIPTitalic_pend_POSTSUBSCRIPTusinga3-parameterlogistic(3PL)model:
Ordinalmodelsmaybeemployedtoaccountforordinalresponsesinitems,suchasthoseinvolvedinpsychologicalassessments.Onesuchmodelistheadjacent-categorymodel,alsoknownastheGeneralizedPartialCreditModel(GPCM)withintheIRTframework.Inthismodel,theso-calledadjacent-categorieslogits,whichrepresentthelogarithmsfortheratioofprobabilitiesfortwosuccessivescores,areassumedtohavealinearform:
Thedevelopersaresupposedtobuildtheirsourcepackagesandsubmitthemtothefirstauthorofthepaperviae-mail.
ThesampleSIAmodulespresentedinthissectionmayserveasaninspirationforthepossibleextensionsofthemainSIAapplication.Someofthesemodulesusetheirdatasetsonly,whileothersallowinteractionwithdatafromthemainSIAapplicationorevengeneratedatasetstobepassedintothemainSIAapplication.
ThisdemonstratesthecapabilitytoextendtheIRTanalysiswithintheSIAapplication,whichcurrentlysupportsonlythetestswithasingleitemtype.Theitem-specificIRTmodelingisprovidedintherespectivetabofthemodule,alongsidesomecustomizedtraditionalitemanalyses.Additionally,themoduleoffersthefunctionalitytocreatebinarygroupingand/orcriterionvariablesfromafactorvariablewithmultiplelevels,whichcanbeutilizedforDIFdetectionwithintheSIAapplication.
TheDIF-CmodulefromtheSIAmodulespackageopensupthecoreanalysisinaninteractiveanddirectlyreproducibleway.Additionally,ThecorrespondingLearningToLearndatasetisconvenientlyaccessiblewithinthemainapplicationand,therefore,canbeanalyzedusingvariouspsychometricmodelsandapproaches.Additionally,DIFdetectionmethodsinthe"DIF/Fairness"tabofferthepossibilitytoemployanobservedscorevariableasthematchingcriterion,whichisthescorefromthe6thgradefortheLearningToLearn.ThisenablesDIF-Canalysisforthisdataset.Moreover,DIF-Cdetectiontootherdatasetsisachievablebyprovidingthe"Observedscore"variableinthe"Data"section.WhiletheDIF-Cdetectionisaccessibleinthemainapplication,theDIF-CmodulefromtheSIAmodulespackageopensupthecoreanalysisofthepaper\citepmartinkova2020dif-cinaninteractiveanddirectlyreproducibleway.Itprovidesastep-by-stepexaminationofbothscores,asummaryoftheDIF-Canalysis,andplotsofICCsforindividualitems.
AnotheraspectofdatacomplexitynotcurrentlyaddressedinthemainSIAapplicationinvolvesratingsfrommultipleraters.Whenmultipleratersareinvolved,theassessmentofinter-raterreliability(IRR)becomespertinent,typicallyanalyzedthroughmethodssuchasvarianceanalysisor,moregenerally,variancecomponentmodels\citepmartinkova2023assessingIRR.
TheIRRmodulewithintheSIAmodulespackageprovidesaninteractivedemonstrationoftheissuesofusingIRRinrestricted-rangesamplesinthecontextofgrantproposalpeerreview.Themoduledemonstratesthatwhensubsetsofrestricted-qualityproposalsareused,thiswilllikelyresultinzeroestimatesofIRRundermanyscenarios,althoughtheglobalIRRmaybesufficient\citeperosheva2021zero.
Asanotherexampleofamoduleresidinginthe"Reliability"tabofthemainSIAapplication,theIRR2FPRmoduleoftheIRR2FPRpackage\citepIRR2FPRprovidesaninteractiveillustrationofthecalculationofbinaryclassificationmetricsfromIRR,providinganestimateoftheprobabilityofcorrectlyselectingthebestapplicants\citepbartos2024irr2fpr.
EduTestTextAnalysismodulefromtheEduTestTextAnalysispackage\citepedutesttextanalysisseekstoprovideatoolforitemdifficultypredictionbasedsolelyontheitemwording\citep[see][fortheunderlyingresearch]stepanek2024.Themoduledoesnotuseanydatafromthemainapplication,nordoesituploadanytabulardata.Instead,itusestextinputfieldsandthedatabaseofseveralitemexamples.ThisisanotherdemonstrationoftheversatilityoftheSIAmodulespertainingtotheinputnature.
Anotherimportantfeaturethatthismoduleillustratesistheusageofcomplexandlargemodelsspanninggigabytesofbinarydata.Oneofthecrucialindependentvariablesinthepredictivemodelisthecosinesimilarityofdifferentitemwordingparts\citepstepanek2024,calculatedemployingtheword2vec\citepword2vecwordembeddingsmodel.Inthemodule,weimplementedamechanismthatcandownloadandcachethecompressedbinarymodelfromtheinternetondemandandutilizeitimmediatelyintheanalysis,thusprovingthatlargeandcomplexmodelsaremanageableintheproposedmodulararchitecture.TheEduTestTextAnalysismodulealsodemonstratestheuseofcompiledC++librariesthattheword2vecpackageiswrapping.
Inthispaper,wehaveoutlinedtheprocessofdevelopingnewadd-onmodulesfortheSIAinteractiveappinRwiththehelpoftheSIAtoolspackage.Wedemonstratedthefundamentalprinciplesandoptionsusingseveralmodulesthatarealreadyavailableandofvaryingcomplexity.InteractiveShinyapplicationshavethepotentialtoexpandtheusercommunity,andwhenRcodeisprovided,theyalsohelpnewcomersinadoptingR.Theplatformweintroducedforadd-onpackages,utilizedintheSIAframework,mayeaseexternalcollaborationandcustomizationofshinyprojectsingeneral.
Tothebestofourknowledge,onlyjamovi\citepjamoviandJASP\citepjaspmaketheiranalysesfullyorpartiallyavailabletobeusedprogrammaticallyfromwithinRconsoleastheybuildupontheunderlyingRpackages(jmvpackage\citepjmvforjamoviandvariouspackagesforJASP),althoughbotharemeanttoberunasastandalonesoftwarewithGUIinthefirstplace.
WeoffertheSIAtoolspackageasaresourcetofacilitateSIAmoduledevelopment.Asimilartoolkitisprovidedinthejmvtoolspackage\citepjmvtools,whichalsoprovidesafewtemplatesandcruciallyusesaproprietarycompilertocreateamodulethatcanbeusedinthejamovisoftwarewithGUI.Ontopofthat,jmvtoolsneedsaspecialJavaScriptruntimeenvironmenttooperate.ThisisalsothecaseforanothersimilarpackagecalledjaspTools\citepjaspTools,whichreliesonanumberofexternaldependenciesaswell.Incontrast,theSIAtoolsworkssolelywithintheconfinesoftheRlanguage.
ThereareseveralaspectsworthconsideringinfutureversionsoftheShinyItemAnalysis,SIAtools,andSIAmodulespackages.TheSIAtoolspackagemaybecomemorerefinedintermsofmoduletesting,building,andsubmitting.Orpossibly,whentheSIAapplicationofferstheautomaticgenerationofPDFandHTMLreports,includingmoduleresultsinthereportmaybefeasible.Furtherautomationofgeneratingthereportsviaacommand-lineenvironmentmayfosterautomation,reproducibilityandreusewithotherpackages.
Whiletheseimprovementswouldincreasetheusefulnessofthecurrentapproach,thepresentedversionalreadyrepresentsavaluableextensionoftheShinyItemAnalysispackage.Webelievethatitsinnovativenatureandpracticalutilityhavethepotentialtonotonlyinspirebutalsoinfluencefutureprojectsinthisdomain.
ThestudywasfundedbytheCzechScienceFoundationproject"Complexanalysisofeducationalmeasurementdatatounderstandcognitivedemandsofassessmenttasks"grantnumber25-16951S,bytheproject"ResearchofExcellenceonDigitalTechnologiesandWellbeingCZ.02.01.01/00/22_008/0004583"whichisco-financedbytheEuropeanUnion,andbytheinstitutionalsupportRVO67985807.TheauthorswouldliketoacknowledgeFrantiekBartoforhelpfulcommentsonthepreviousversionofthemanuscript.