A statistical method for the evaluation of projectile dispersion

C.A.Rabbath,D.Corriveau

Defence R&D Canada,2459 De La Bravoure Rd.,Quebec,QC G3J 1X5,Canada

A statistical method for the evaluation of projectile dispersion

C.A.Rabbath,D.Corriveau*

Defence R&D Canada,2459 De La Bravoure Rd.,Quebec,QC G3J 1X5,Canada

A R T I C L E I N F O

Article history:

Dispersion

Small-caliber projectile

Hypothesis test

Con fidence intervals

Variance

Precision trials

Populations

Mann barrels

Algorithms

As part of a research program,an extensive study on the dispersion characteristics of eight different 7.62×51 mm ammunition types was conducted.The paper presents the main steps in the experimental and analytical process carried out to evaluate,namely to measure and compare,the dispersion characteristics of the ammunitions;namely,(1)identify the number of rounds to fire in the trials,(2)establish a test plan and the setup for the precision trials,(3)fire the rounds,following an established protocol for the experiments,(4)collect the impact points,and measure the performance through statistical measures,(5)perform a statistical analysis of dispersion applied to the results obtained in the trials,and(6) conclude on the ammunition characteristics.In particular,the paper proposes a statistical method to evaluate the precision of ammunitions fired with precision(Mann)barrels.The practical method relies on comparison of con fidence intervals and hypothesis testing on the standard deviation of samples, namely the impact points.An algorithm is proposed to compare the variances of two or more populations of ammunitions.

Crown Copyright©2017 Published by Elsevier Ltd.This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).

1.Introduction

The paper presents the main steps of the experimental and analytical process carried out to evaluate the dispersion characteristics of eight different 7.62×51 mm ammunitions.Brie fly, through repeated firings of a number of rounds,complying with a uniform modus operandi,and by means of a post-trials statistical evaluation of the collected impact points,one expects to be able to distinguish among the precision obtained with the various ammunition types.However,several questions arise.To what extent can a distinction be made?Can one establish,with a limited number ofrounds,thatone ammunition type is superior to another in terms of its statistics of dispersion?Is the difference in performance between two different ammunitions simply due to the limited number ofrounds fired?Whatis the con fidence levelin the evaluation ofthe dispersion?Very few answers can be found in the literature.These questions are addressed in the paper.

There are eight 7.62×51 ammunition types to test.The ammunitions are labeled and succinctly described as follows:A1(open tip,boattail,168 gr),A2(Gilded metaljacket,148 gr),A3(Fullmetal jacket boat tail,170 gr),A4(Full metal jacket,176 gr),A5(Gilded metal,open tip,boat tail,175 gr),A6(open tip,boat tail,175 gr),A7 (open tip,boat tail,175 gr),and A8(175 gr).The rounds are fired with three types of 508 mm long precision(Mann)barrels:barrel with a twist of1 turn in 254 mm or 10 inches(twist of1/10),barrel with a twist of 1 turn in 285.75 mm or 11.25 in(twist of 1/11.25), and barrel with a twist of 1 turn in 304.8 mm or 12 in(twist of 1/ 12).Barrel twist determines the roll rate of the gyrostabilised projectile.

The raw measure ofdispersion,prior to a statisticalanalysis and processing of the results,is the observation of the grouping of the impact points on target.Ifallrounds are fired toward the same aim point,assuming that the same equipmentis used for allthe rounds, then the proximity,or the grouping,of the impact points obtained after repeated firings is the first observation one can make on the level of performance of given ammunitions,Mann barrels,and shooter-target ranges[1-3].

A relatively small grouping of the impact points generally indicates a relatively high level of consistency(or precision)[1]. Assuming that one fixes,and maintains constant,the firing conditions and the shooter-target range,that wind speeds are below an acceptable threshold,and that the data acquisition system is tuned uniformly for all tests,one thus has the conditions for a fair comparison.Yet,the comparison has to be made with care,using the appropriate statistical tools and methods,and with the correct interpretation of the tests,especially if the patterns of the groupings appear,at first glance,to be relatively similar for the differentammunitions.

A statistical evaluation is therefore necessary to infer precision characteristics for an ammunition type,and to distinguish among the characteristics of the different ammunition types.One collects measurements,and then concludes on the ammunition characteristics in the contextofthe trials.This is the evaluation considered in the paper.To conduct the evaluation,a statistical method for dispersion analysis is proposed.The practical method relies on comparison of con fidence intervals and hypothesis testing on the standard deviation of samples,namely the impact points.A relatively simple iterative algorithm is proposed to compare the variances of two or more populations of ammunitions.

The paper is divided as follows.Section 2 presents the calculation ofthe number ofrounds to fire in trials,the testplan and setup for the precision trials,and the actualprotocolfollowed during the experiments.Section 3 provides the main results collected during the trials in the form of statistics,and measures ofperformance,on projectile impactpointlocations.Section 4 focuses on the statistical analysis of dispersion using the results obtained in the trials.A statistical method that relies on hypothesis testing and on wellknown tests on the standard deviation of samples,and a relatively simple iterative algorithm that enables a comparison of variances for two or more populations are proposed in Section 4. Finally,conclusions on the ammunition characteristics are stated in Section 5.

2.Pre-trials calculations and preparation

Prior to carrying out the precision trials for the eight ammunition types,and for the three Mann barreltypes,a preparation phase is needed.This is the topic of this section.

2.1.Number of rounds to fire

The number of rounds must be such that the measurements expected from the trials are statistically relevant;in other words, there must be enough rounds fired to carry out the evaluation.Yet, to control costs and the duration of the trials,it is clear that one wants to fire a number ofrounds that is reasonable.Thus,one seeks the minimum acceptable number of rounds.

To set the number of rounds,one relies on(1)a generalized full factorial design(GFFD)method[5,6],and(2)sample size determination methods(SSDM)that seek to constrain standard deviation differences with that of the population[6-8].

In this paper,a population is the set of all impact points associated with an ammunition type,a range and a barrel type.A sample is simply a subset of the population.Trials measurements constitute a sample.

2.1.1.Mann barrel selection

There are three barrel types(twists of 1/10,1/11.25 and 1/12), and three copies of each type of barrel.There is therefore nine barrels in total.One would like to select a single barrelamong the three copies of the same type,and do so for each barrel type.To select the barrels,a number of rounds are fired with the nine barrels using only ammunition A1,at a range of 200 m,assuming that the distribution of the impact points is normal,and assuming that each firing is an independent event.Then one uses the selected barrel with twist of 1/10,that with twist of 1/11.25 and that with twist of 1/12 to carry out the remainder of the precision trials,to fire all eight ammunition types.

GFFD is run in MINITAB®[6]with a single factor(barreltype,the independent variable),three levels(barrel twists),power p set to 0.9,signi ficance levelαset to 0.05,maximum range of differences in the dependentvariable,such as impactpointlocations,setto d= 0.4 cm[4],and standard deviation(SD)of 1 cm.The latter values are selected from past experiences.Obviously,the estimated SD value may be different from the one calculated with the trials data, once the data is collected.Yet,a value must be set to serve as an approximation and giving a general idea of the number of rounds required.Furthermore,the value of 0.4 cm may be reduced,at the costofa signi ficantincrease in the sample size required,which is to be avoided here.The resulting number of rounds to fire is 160 per precision barrelcopy.With this number of rounds,one may detect differences of 0.4 cm in the main effects with a power value of 0.9, for three copies of barrels of the same type.

Power(typically close to 1)is the likelihood that a signi ficant difference(or effect)is identi fied when one truly exists[5].The signi ficance level(typically small)can be interpreted as the risk of obtaining false differences in the results,on the hypothesis[4,5].

The SSDM of[7,8],establish the number of rounds to fire such thatone may bound the difference between the calculated SDofthe sample with that of the population to a desired value.There is typically a levelofcon fidence associated with the bound.Here,one sets a 95%con fidence level.The SSDM indicate that with 160 rounds,one obtains a SDofthe sample within between 10%and 15% of the SD of the population.It should be noted that the contour plots of[8]as wellas the other SSDMexhibita sharp increase in the sample size if one requires errors below 10%in SD(sample versus population statistics).

2.1.2.Evaluation of precision

There are two factors of in fluence:barreltype and ammunition type.There are 8 levels for the ammunition type factor.There are 3 levels for the barreltype factor.Recallthatthere is a single copy ofa barrel type and there are three barrel types(twist),following the barrelselection process.The values for the key GFFDparameters are set to d=0.1，0.2，0.3，0.32，0.35，0.40，0.50 cm, p=0.9, α=0.05,and SDσ=1 cm.

The results of the application of GFFD in MINITAB®[6]are shown in Fig.1.From the figure,one may note:(1)for a fixed number of replicates(Reps),which correspond to the number of rounds per ammunition per barrel,a reduction in maximum difference is associated with a reduction in power,(2)for a fixed level of power,reducing d results in a nonlinear increase in the number of replicates.One may select a value of 120 rounds per ammunition per barrelfor an ammunition precision evaluation with maximum detection ofmain effects of0.32 cm.No formalrules are used in this selection.

With the SSDM of[7,8],one obtains an error between 10%and 15%between the population SD and that calculated with a sample of 120 rounds for a given ammunition type and precision barrel assuming(1)normal distribution of the impact points,(2)and a planned value of 1 cm for the SD.

2.2.Test plan

Two in-service ammunitions,A1 and A2,are tested in part1.Part 2 pertains to trials with the six candidate ammunitions,A3 to A8.

2.2.1.Part 1 of trials

2.2.1.1.Target and range.A target is setup indoors at a distance of 200 m from the muzzle of the barrel.A target is setup outdoors at a distance of800 m from the muzzle of the barrel.A video camera is positioned in proximity of each target,pointing at the target,in order to identify and collect the impact points associated with each projectile.

2.2.1.2.Ammunitions and barrels.For the part of the trials pertaining to the precision barrelselection,160 rounds are fired with ammunition A1 in addition to warmer rounds.Ammunition A1 is fired with the 9 barrels available,labeled as follows:Barrels 1 to 3 (with twist of 1/10),barrels 4 to 6(with twist of 1/11.25),and barrels 7 to 9(with twist of1/12).The precision(Mann)barrels are installed on an M852 chamber mounted on a V-block and Springfield 1903 mount(bolt action).There are 2 warmer rounds per series of 15 rounds.The ammunitions are conditioned at+21°C. After initialtrials,a single copy ofeach barrelcon figuration(twists 1/10,1/11.25,1/12)is selected and the remainder of the trials is conducted with the selected copies on a bolt-action mount.

2.2.1.3.Tracking radar.Tracking radar measurements are obtained for each round(warmers included)in order to determine the muzzle velocity,the trajectory(flight behaviour)as wellas the drag force on the projectile,and the velocity close to the target.

2.2.1.4.Aiming.Aiming on the target is performed with the help of a boresight lens.The aiming pointis veri fied every 5 rounds fired to make sure that the weapon has not moved.

2.2.1.5.Atmospheric conditions.The atmospheric conditions at the time of firing(temperature,pressure)are recorded.The trials are done atambienttemperature.There is no constrainton the outdoor air temperature.To record the data associated with a round, effective wind speeds(effective meaning orthogonal to the line of sight)must be below 4 m/s.A meteorological station is placed at the shooter location,at mid-range between shooter and target,and at the target.

2.2.1.6.Equipment.The following equipment is used:video(electro optical)camera at the target,a tracking radar,ammunitions A1 and A2,temperature and pressure measuring devices at three locations,and boresight lens.

2.2.1.7.Measurements.The following quantities are measured and recorded during the trials:impact location ofprojectiles on targets, muzzle velocities,velocity along trajectory(flight behaviour), especially close to target,and atmospheric conditions atthe time of firing.

Data are recorded in tabular format.Table 1 presents a summary of the tests for part 1 of the trials.

Precision barrelselection trials correspond to Tests 1 to 9 in the preceding table.One copy of each precision barrel type/con figuration is selected after the initial tests shown in Test 1 to Test 9.

The remainder of the tests use one copy of each barrel con figuration(twist1/10,1/11.25 and 1/12),namely Tests 10 to 21.For Test 10 to Test21,pre-trials statistics indicate a minimum number of120 rounds per barreltype per ammunition type.

2.2.2.Part 2 of trials

The following elements are identical to those used for part 1: target and range,a selected copy ofeach barreltype,tracking radar, aiming,atmospheric conditions,equipment,and measurements. The difference with part 1 lies in the ammunitions tested,for part 2 of the trials the tests are done on ammunitions A3 to A8.

Again,data are recorded in tabular format.Table 2 presents a summary of the tests for part 2.

2.3.Trials setup

The schematics of the experimental setup are shown in Fig.2. Pictures of the setup are given in Fig.3 and Fig.4.

2.4.Firing protocol

The firing protocolcomplies with the NATO modus operandi[9]. A method employed to reduce the bias due to the environment(i.e. wind,humidity,temperature)is to alternate between series(tests) of the different ammunitions;that is,to switch from one type of ammunition to another after each test.

3.Experimental results

The measured impact points and the statistical measures of dispersion applied to the raw data obtained in the trials are presented as dimensionless quantities.

Impactpointmeasurements are available as coordinates in the x and the y axis of the Cartesian plane on the target.The raw impact points are centered at the origin,at the mean point of impact.

It should be noted that the projectiles are gyroscopically stablewhen fired from all three types of Mann barrels.At 15 degrees Celsius,the gyroscopic stability factor for the 175 gr projectiles is between 1.8 and 2 when fired from the barrels having a 1/12 twist. For the lighter projectiles and with the faster barrel twists,the gyroscopic stability factor is larger than 1.9.

Table 1 Summary of tests for part 1 of the trials.

Table 2 Summary of tests for part 2 of the trials.

3.1.Basic measures of performance

The standard deviation(SD)σof a sample of n measurements, with mean x and each impact point measurement denoted as xi,is calculated as[4]

The effective standard deviation(ESD)is given as

The maximum spread along x and y between the origin and any impact point is labeled as M.The maximum distance between any two impact points is denoted as D.

3.2.Mann barrel selection

Impact points collected in tests 1 to 9 of part 1 of the trials are evaluated.The grouping of the impact points obtained with each barrel is measured.If a barrel offers signi ficant dif ficulties in its operation,affecting the safety ofthe personneland the integrity of the equipment for example,it is simply discarded during the trials. Measures obtained with the raw data collected with the firings of ammunition A1 at 200 m are presented in Table 3.Measuresσx,σy,M and D are calculated with the raw data(impact points)of part 1. The results obtained with barrels#2 and#4 are notshown because of the occurrence of practical dif ficulties that jeopardized the precision ofthe rounds;namely,signi ficantmovementofthe barrelon the V block mount,and back attachment that unscrews after severalseries of rounds fired.

Table 3 SD,M,and D as fractions of the maximum value for each column.

Table 4 Barrel selection matrix.

Table 4 provides an overview of the assessment ofperformance using the results of Table 3.An entry in Table 4 is a barrelthatoffers the smallest value for a given performance measure(column) among the barrels having the same twist.The lastcolumn of Table 4 indicates the barrels selected for the remainder of the trials.It is proposed in this paper to select the barrel that receives the most votes(the smallest performance measures)among the barrels having the same twist(on the same row).

3.3.Dispersion of impact points at 200 m range

Impact points collected in tests 10 to 15 of part 1 and tests 1-3, 7-9,13-15,19-21,25-27,and 31-33 of part 2 of the trials are evaluated.The results pertain to the impact points obtained with the eight ammunitions at the range of 200 m and with the barrels selected according to Table 4.The dimensionless results presented in Tables 5-7 are obtained as follows.From the raw data,calculate the measures ofdispersion.Identify the maximum value for a given measure(column)over the three tables.Express all the other values,for the same column,as fractions(ratios)of the maximum value.Do the same for all6 measures(columns)of Tables 5-7.A2 is clearly the ammunition offering the worst performance according to all6 performance measures.

Fig.5 shows a subset ofthe impact points expressed in terms of an unknown unit of length d∈ℝ.

As the first step in the precision analysis,one simply calculates the dispersion characteristics of the samples,as presented in Tables 5-7.However,one cannot conclude on the populations from such calculations.Yeta basic evaluation may serve as a first take on the assessment of precision,a necessary step towards a more comprehensive analysis.SD determines the precision.In general, precision improves with a smaller value of SD.Measures M and D should be used with care as they may pertain to speci fic data points, such as outliers,that are not representative of population dispersion.One may readily assess the smallest SD measures from the tables:(1)comparing the SD obtained with the various ammunitions for a given barrel,and(2)comparing the SD obtained for the three different barreltypes for a given ammunition type.From the tables,A6 offers the smallest value of SD,whereas the barrelwith twist 1/11.25 offers the smallest SD for most of the ammunitions.

Table 5 Dispersion characteristics of ammunitions fired at 200 m range with barrel3(twist 1/10).

Table 6 Dispersion characteristics of ammunitions fired at 200 m range with barrel5(twist 1/11.25).

Table 7 Dispersion characteristics of ammunitions fired at 200 m range with barrel8(twist 1/12).

3.4.Dispersion of impact points at 800 m range

Impact points collected in tests 16 to 21 of part 1 and tests 4-6, 10-12,16-18,22-24,28-30,and 34-36 of part 2 of the trials are evaluated.The results pertain to the impact points obtained with the eight ammunitions at the range of 800 m.The dimensionlessmeasures of performance are presented in Table 8 and Table 9, obtained following the process discussed in Section 3.3.Fig.6 shows a subset of the impact points expressed,again,in terms of an unknown unit of length d.

It is clear from the tables that some of the 800 m data are missing.The reason is simple:the results of the 200 m trials of Section 3.3 indicate that the 1/10-twist precision barrel offers the worst precision among the three types of barrel.Therefore,to manage costs and to provide timely results,rounds have not been fired with the 1/10-twistprecision barrelatthe 800 m range facility besides A1.The few results obtained with A1 at 800 m are not shown for brevity.Furthermore,A2 rounds have not been fired at the 800 m range with the 1/11.25-twist barrel.This saves time and reduces the cost of the trials.It is clear that A2 exhibits the worst precision among the ammunitions tested.Thus,a decision was made during the trials to stop firing A2 at 800 m given the poor results obtained at 200 m.From Tables 8 and 9,ammunitions A1 and A2 provide larger SD than the other ammunitions for both barrel twist types,and barrel with twist 1/12 results in general in the smallest SD,except for ammunitions A3 and A6.Yet,the difference in SD for most ammunition types is relatively small.

It should be mentioned that the averages of the recorded projectile speeds at the 800 m target lie between 300 and 400 m/s.Standard deviation on the velocity at the target range from 3 to 14 m/s.Projectile A1 exhibits the smallest average velocity at the target.With such range of values,it can be observed that the projectiles do not go into the subsonic regime.

Table 8 Dispersion characteristics for rounds fired at 800 m with barrel 5(1/11.25 twist).

4.Statistical analysis of dispersion

One of the original contributions of the paper is the statistical method proposed to evaluate the precision of the ammunitions fired with precision(Mann)barrels.The method allows inferring population characteristics from the samples obtained in the trials.It is assumed that each impact point is obtained independently,with coordinates along the x and the y axes on the target.

4.1.Normality of impact point distributions

Normality is particularly important when developing a statistical analysis of the precision of ammunitions;for instance,when using methods of hypothesis testing[4]and procedures for con fidence intervals on the SD of the distribution of the impact points [10].Therefore,it makes sense to test whether the samples form normaldistributions,with some degree of con fidence.

Table 9 Dispersion characteristics for rounds fired at 800 m with barrel 8(1/12 twist).

The following quantitative tools are applied to the trials data: the Anderson-Darling(AD)test[11-13],the Kolmogorov-Smirnov (KS)test[14],and the Ryan-Joiner(RJ)test[15].The AD,KS and RJ tests are applied to the impact points obtained along x and y, separately,for each testofthe trials;that is,for each row of Tables 1 and 2.Withoutgoing into the details,the result ofeach testis either a normaldistribution,not a normaldistribution,or an inconclusive test(due to a borderline value for a metric,such as the p value[10], associated with a test).Normality tests are not applied to rows of Tables 1 and 2 with insuf ficient data,signi ficantly below the planned rounds.

To decide on the normality of the distribution for each row,in x and y,a simple voting strategy is used with equal value given to each ofthe three tests(AD,KS,RJ).The conclusion ofthe normality test is that there is not suf ficient evidence to reject the hypothesis that the distributions associated with the collected samples are normalalong the x and the y axes.

4.2.Comparison of con fidence intervals on standard deviations

The next step in the analysis of precision is to determine,with some level of con fidence,intervals within which the populations' SD are expected to lie from the knowledge of the characteristics of the samples.This is the concept of con fidence intervals(CI)as applied to the calculated SD[16].

A CI is a range,or interval,of values that is likely(namely,to a certain degree of con fidence)to contain the value of an unknown population parameter,here the SD[16].95%CIare widely accepted; namely,with a 95%probability,the SDofthe population is expected to fallwithin the CIfor the SD ofthe sample[5].Thus,to determine whether there is a difference in dispersion between any two ammunition types,for a given range and barrel,one simply looks for a separation among the CIon SD.

The approach proposed for this part ofthe analysis is as follows. The CIon the SD of the samples are calculated with MINITAB®[6] for each barrel/range and for the impact points collected with the eightammunitions,and along x and y.The CIobtained with barrels 5 and 8 are plotted against ammunition type in Fig.7 and Fig.8,as dimensionless quantities,and compared.In the figures,a CI is a verticalline with upper and lower bounds(denoted as letter x or as the symbol of a triangle).The ammunition populations that are likely to have the smallest standard deviations are those for which the CIoverlap,or are relatively close to,the smallest CIstrip in the figures.Each figure contains two strips:one for the smallest CI along the x axis and one for the smallest CIalong the y axis.The smallest CIwidth or strip is associated with only one ammunition type:the ammunition type with the smallest con fidence limits in its standard deviation's 95%CI.Any part ofa CIthat falls within the smallest CI is also considered to probably have,to some level of con fidence,a standard deviation that is the smallest among the ammunitions considered in this study.Ifthe CIof two ammunition types do notoverlap,it is highly likely that the standard deviations oftheir respective population are different.Ifthey are in proximity, however,a conclusion is dif ficult to draw.Here we propose to keep such non-overlaping,yet close,intervals.An example is A4 for 200 m range,barrel5.Note that SDfor A2 is outside the scope ofthe graph for 200 m range,barrel5.The conclusions ofthe comparison are given in Table 10 for the x axis only,for brevity.Similar trends are observed for the y axis.

Table 10 Ammunition types likely to have the smallest SD using the comparison of CImethod (x axis).

The so-called Bonferroni con fidence intervals are typically wider than the 95%CI[17].The use of Bonferroni CI is a conservative alternative.Yet,if one calculates Bonferroni CI for the data presented in this paper,the same ammunition types as those identi fied in Table 10 are found to be the ones likely to have the smallest standard deviation.This is so since the ammunitions with non-overlaping,yet relatively close,intervals are kept in the analysis,along with those overlaping intervals.

4.3.Hypothesis testing and comparison of two ammunition populations

Hypothesis testing(HT)and statistical tests may help clarify potentialambiguities with an analysis based on a comparison of CI on the SD of the samples,as discussed in Section 4.2.The method proposed in this section enables testing the hypothesis of equivalence of variances,and hence SD.The basic assumptions made on the samples and populations for HT to bear value are the following: (1)the samples are independent and random,and(2)the populations are normal.The normality of the data is discussed in Section 4.1.

A hypothesis is a statement about the numerical value of a population parameter[4],which is either the SD or the variance in this paper.Here,one develops a hypothesis on the dispersion characteristics of two populations,namely on the ratio ofthe SD of the two populations,and tests the hypothesis using the dispersion characteristics of the samples.With such an approach,onecompares the dispersion associated with one population against that of another population.

HT generally consists of the following ordered steps[5]: formulate the hypothesis,choose the signi ficance level,select an appropriate test,calculate the test statistic,make the test decision, and interpret the results.These steps are followed in the approach proposed in the paper.

4.3.1.Formulation of hypothesis and selection of signi ficance level

Two statements are made for a given test:the null hypothesis H0,and the alternative hypothesis Ha.The null hypothesis is the status quo.It is accepted unless the data provide evidence that it is false[4].H0is a claim on the value of SD.The alternative hypothesis, Ha,is obviously different from H0.

With the knowledge of H0and Ha,one elaborates a criterion,or a test,to either reject H0in favor of Haor not reject H0.The test statistic is computed from the data of the sample that is used to decide between H0and Ha.Before presenting the tests,one should be aware of errors that may occur in the assessment of the hypothesis.The so-called Type Ierror[5]is the error that takes place when H0is true but,based on the sample,one rejects it in favor of Ha.The probability ofcommitting a Type Ierror is denoted byα,and is de fined as[5]

One can select the levelof risk,or the signi ficance levelofa test, α,ofmaking a Type Ierror.Values forαare typically 0.01,0.05 or 0.1. Hereα=0.05 is selected.

Another possible error is the so-called Type II error[5].This error occurs when H0is actually false,butbased on the information contained in the sample one does not reject H0.The probability of committing a Type IIerror is denoted byβ,and is de fined as[5]

According to[10],the power ofa statisticaltestis the probability that the test rejects H0when H0is indeed false and is given as

4.3.2.Test selection

To compare variances of two ammunition populations,the socalled F-test is used[18].Consider two populations,one having standard deviationσ1and the other having standard deviationσ2, both taken along the same axis on the Cartesian plane,either x or y. Hypothesis H0states that the ratio of the variances of the populations is unity,or

whereas an alternative hypothesis states that one population variance is smaller than the other,and is expressed as

Since one has access to the statistics of the samples,one calculates the test statistic given as

where siis the calculated standard deviation for the sample of population i.Such terminology is only used in this section of the paper.The statistical test is therefore a comparison between two populations,and relies on the statistics of the samples.With Hagiven in(7),one has a one-tailed test[4]since the alternative hypothesis suggests a particular direction with respect to H0.The test statistic F is used to decide between the null and the alternative hypotheses[4].One needs to de fine the values of F for which the nullhypothesis is rejected in favor of Ha.Such values of F form the rejection region[4].When the ratio falls into the rejection region, one refutes the null hypothesis,and the variances differ.The rejection region is such that the probability isαthat it willcontain the test statistic F when H0is true.

The minimum F-value ofthe rejection region is at the tailof the F distribution[4]and is labeled Fc.One rejects the nullhypothesis when F＞Fc.The F-distribution is the sampling distribution of the ratio of Equation(8).Itstrongly depends on the sample sizes for the two populations[4].To calculate Fcone uses the sample size and locates Fcfrom F-distribution tables,as available for instance in Ref.[4].For the one-tailed test,the approximate rejection region is obtained from Ref.[4]as F＞1.352 forα=0.05.

To complementthe HT with the F-test,itis proposed to use the p value as a second metric to determine whether to reject the null hypothesis.Recall that the p value is the smallest level of signi ficance that would lead to rejection of the null hypothesis with the given data.When the p value is smaller than the preset value of a, one rejects the nullhypothesis[4,5,10].

4.3.3.Test statistic calculation

With eight different populations,ideally one carries out NFtests,to evaluate every combination,namely every ammunition pairing,which amounts to NF=8!/(2!(8-2)!)=28.However,one can leverage the results of Section 4.2,and test only those ammunition types likely to have the smallest SD,as identi fied in Table 10 for example(for the x axis only);therefore reducing the number of computations.To do so,one takes every pairing of samples identi fied and calculates their F statistic.

In practice,calculating the F statistic of Equation(8)and the p value with MINITAB®is done with the lines ofscriptshown in Fig.9. In the figure,＜name_column_data1＞and＜name_column_data2＞correspond to the names ofthe two samples(impactpoints for two ammunition types)being compared as available in column data format in MINITAB®.Con fidence of 95 indicatesα=0.05.Parameterμis set to 1 for the Haof Equation(7).

A subset of the calculated statistics is provided in Table 11 and Table 12 for the speci fic cases ofimpact points obtained with barrel 5 at ranges of 200 m and 800 m.These two tables are suf ficient to present the kinds of results obtained with the proposed approach. A non-zero entry in the tables gives the result of the F statistic and the p value in the format(F,p)for comparison ofammunition types given by the row-column pairing.There is no F statistic calculated in empty cells.Such a case may arise when calculations are deemed unnecessary or when there is a lack of data.For this speci fic case of impact points(barrel 5 at ranges of 200 m and 800 m),one may refer to Figs.7 and 8,and Table 10 to conclude on the ammunition pairings considered for F-test statistics.It is important to note that for each F-testcalculation the larger ofthe two standard deviations is placed at the numerator position.

Table 11 F statistic and p value in format(F,p)for 200 m range and precision barrel5 along x (in the white background)and along y(in the grey background).

Table 12 F statistic and p value in format(F,p)for 800 m range and precision barrel5 along x (in the white background)and along y(in the grey background).

4.3.4.Test decision and interpretation of results

Once the statistics have been calculated,one compares(1)the ratio F with the approximate rejection region bound,namely one assesses if F＞1.352,and(2)the p value with the value ofα=0.05 (to check if p＜α).In this paper,if both conditions are met,the null hypothesis is rejected,in favor of Hagiven by Equation(7).

The conclusions of HT on paired populations are provided in Table 13.To be ofany practicalvalue,the proposed approach draws conclusions fromtests on pairings thatare coherent(yield the same conclusions)along both the x and y axes.In such a context,one may draw conclusions only for the following results obtained at 200 m: (1)for the 1/11.25-twist barrel,one favors the hypothesis that the populations of A5 and A4 have larger variance than the population of A6;and(2)for the 1/12-twist barrel,one favors the hypothesis that the populations of A1 and A6 have larger variance than the population of A7.

The obvious limitations ofHT and F-testare thatthe comparison is con fined to two populations at a time,and that the conclusions on the populations likely to have the smallest SD are dif ficult to draw.

4.4.Multi-population variance testing

The algorithm proposed to compare variances among several populations is illustrated in Fig.10.This approach builds upon the methods proposed in Sections 4.2-4.3.

4.4.1.Population elimination process

Here,the null hypothesis states that all the populations have equalvariance;that is,they are homoscedastic[11].The alternative hypothesis states that a population has a different variance than another population for at leastone pairing ofthe populations in the set[18].For given range,precision barrel,and either along the x or the y axis of the Cartesian coordinate system,one begins the analysis with the entire set S of populations;that is,the 8 ammunition types.Yet,this set can be reduced by eliminating ammunition types that are not likely to exhibit the smallest value in standard deviation.This process ofelimination is done even though it is true that reducing the set of ammunition types considered in the analysis affects the results of the tests.This statement is evidenced by the theory behind the statistical tests[19-22]:SD and variance intervals are function of several variables,including the number of populations considered in the analysis and the sample size,to name a few.Therefore,one must keep such information in mind when concluding on the results of the tests.

MINITAB®offers three methods for multi-population variance testing:(1)test of multiple comparisons[6,22],(2)Levene's test [6,20,21],and(3)Bartlett's test[6,19].It is not the intent of the paper to go into the theoreticaldetails ofthese methods.As is done in the method presented in Section 4.3,the p value for the test, either the test of multiple comparisons,Bartlett's test or Levene's test,is calculated and assessed against the value ofα.

With reference to Fig.10,to startthings off,one assumes thatthe set~S is empty.Multi-population variance testing is done on the set. When p＜α,the nullhypothesis is rejected,and hence at least one ammunition type has a variance which is different from the others. One removes from the set the ammunition type whose CI in its variance,or SD,is the farthest from the others.This is best donegraphically,through MINITAB®,albeit the output of the script provided in this section may be used for such an assessment.The test is repeated.After one iteration,the test is carried out on a smaller set of ammunition types.

Table 13 One-tailed tests with rejected null hypothesis.

The test is done again and again untilone cannot reject the null hypothesis,namely until p＞α.The dispersion of the ammunitions present in the last tested set are then said not to be distinguishable from one another,and are considered statistically equivalent in terms of their precision.When p≈α,the test is inconclusive.

The test results may differ for the same ammunition samples along x and y for a given range and a given precision barrel.One should consider only those ammunition types that,atthe end ofthe process,agree in both x and y for the given range and precision barrel.

Partofthe algorithm of Fig.10 requires specialattention;that is, the update of set~S.Removing a particular sample in the test may not be as straightforward as indicated in the figure.For example, there may be several samples whose CI appear to be relatively distant from the others,or there may be several overlapping intervals.In such cases,one should identify every possible subset of ammunition types(remove the different ammunition types),and carry out the tests for such reduced sets.Importantly,in the end, the process should lead to the set with the largest number of ammunition types(whose values are in the lower end of the spectrum of SD,obviously)for which the nullhypothesis cannot be rejected.

4.4.2.Statistical tests

Brie fly,the test of multiple comparisons may be used for any number of populations.The intervalfor the variance of a sample is compared against that of another sample.This is done for every possible pair of samples.The calculation of the interval follows a procedure introduced in Ref.[22].A p-value is computed for each pairing.The minimum value among allofthe p-values calculated is obtained and compared against a to reject or not the null hypothesis of equality of variances.Graphically,the null hypothesis is rejected if and only if at least one pair of the calculated intervals does not overlap[6].

The statistic calculation carried outwith Levene's testis given in Ref.[20].The test is known to offer some level of robustness to relatively small sample sizes.A single p-value is computed for the set of populations tested and compared againstαto reject or not the null hypothesis of equality of variances.

Bartlett's test statistic[19]gives some sort of weighted arithmetic and geometric averages ofeach sample variance,as explained in the help utility of MINITAB®.The likeliness that the variances of the samples are not equal increases with the difference in the averages[6].Once again,a single p-value is computed,and compared againstαto reject or not the null hypothesis of equality of variances.Bartlett's test is known to be relevant for normally distributed data[6,19],as is the case here.

One could choose to apply all three tests at each iteration,and have either consensus or most votes of successful tests to determine if the nullhypothesis is rejected.

4.4.3.Implementation

The multi-population variance test is implemented on the MINITAB®prompt by means of the script shown in Fig.11 for n column data(n samples).Con fidence of 95 indicatesα=0.05.It should be noted from Fig.11 thatthe UseBartlett line is inserted only when one wants to use speci fically Bartlett's test.If such a line is omitted,the testofmultiple comparisons and Levene's testare both carried out.Once the computations are completed,the results of the tests are displayed on the MINITAB®prompt.The user can then eliminate the appropriate samples,if need be,and run again the script with the relevant samples.The process continues until one cannot reject the nullhypothesis.

4.4.4.Results

The main results obtained with the proposed multi-population variance testing are shown in Table 14.In the table,the ammunitions in boldface type are those which are part of the set of ammunitions having indistinguishable,and the smallest,variance along both x and y axes.This is in the sense that one cannot reject the null hypothesis of equal variances among the populations in those sets ata signi ficance levelofα=0.05.A2 is clearly notpartof any set.For the 200 m range,and for the three barreltypes,A5,A6, and A7 offer the smallest SD in the sense of the proposed multipopulation variance assessment.For the 800 m range,and for the 1/11.25-and 1/12-twist barrel types,A3,A5,and A7 exhibit the smallest SD.

4.5.Summary of statisticalanalysis

The statistical analysis of dispersion relies on three steps:(1) comparison of CIon SD,(2)hypothesis testing to compare pairs ofammunition populations,and (3)multi-population variance testing.With steps 1 and 3,one is able to reduce the initial set of ammunitions to a subset expected to contain the ammunitions with statistically indistinguishable variance along both x and y axes, and likely having the smallest variance.Fig.12 and Fig.13provide a summary of the results obtained with steps 1 and 3.With hypothesis testing and the F-testapplied to two populations at a time (step 2),however,it is dif ficult to conclude on ammunition dispersion characteristics.As discussed,such a step in the analysis is helpfulto determine with some levelofcon fidence the likeliness that one ammunition type has a smaller or a larger SD ofits impact points as compared with any other ammunition type.Furthermore, HT is a cornerstone for the approach proposed in step 3.

5.Conclusions

The paper presents the main steps in the experimental and analytical process carried out to measure and compare the dispersion characteristics of eight different ammunitions.Through carefully planned trials,firings of a number of rounds at two different ranges and with three types of Mann barrels,and by means of a post-trials statisticalevaluation of the collected impact points,one is able to reduce the initial set of ammunitions to a subset expected to contain the ammunition types with statistically indistinguishable variance along both x and y axes on target,and likely having the smallest variance(dispersion characteristics). However,one is not capable of identifying a single ammunition type as having the best dispersion characteristics according to the performance measures presented in the paper,for the number of rounds selected,and for the ranges and the conditions ofthe trials. The number of rounds fired is limited mainly due to costs.One therefore seeks to obtain the best possible dispersion analysis with a constrained number of rounds.

The paper proposes a statistical method to evaluate the precision of ammunitions.The practicalmethod relies on a comparison of con fidence intervals on standard deviations,on hypothesis testing on the standard deviation of the samples,or impact points, and on multi-population variance testing,which is achieved through a population elimination process.The method,which allows concluding on population dispersion from the analysis of the samples,is implemented as a setofrelatively simple algorithms on commercialsoftware.A summary ofthe conclusions ofthe analysis is given in Figs.12 and 13.

[1]Longdon LW.Textbook ofballistics and gunnery,vol.1.H.M.Stationery Of fice; 1984.

[2]Driels MR.Weaponeering-conventional weapon system effectiveness. American Institute of Aeronautics and Astronautics;2004.

[3]Accuracy and precision, https://en.wikipedia.org/wiki/Accuracy_and_ precision[Accessed online:18 July 2013].

[4]McClave J,Sincich T.Statistics.12th ed.Pearson;2013.

[5]Gupta BC,Guttman I.Statistics and probability with applications for engineers and scientists.Wiley;2013.

[6]MINITAB,http://www.minitab.com[Accessed 5 January 2017].

[7]Triola MF.Elementary statistics.12th ed.Pearson;2014.

[8]Greenwood JA,Sandomire MM.Sample size required for estimating the standard deviation as a per cent of its true value.J Am Stat Assoc 1950;45(250):257-60.

[9]NATO Army Armaments Group(NAAG).Multi-calibre Mopi for 5.56 mm,7.62 mm,9 mm and 12.7 mm ammunition.PFP(NAAG-LCG/1 SG/1)WP.2010. p.0002.NATO/PFP UNCLASSIFIED,December 2010.

[10]Montgomery DC,Runger GC.Applied statistics and probability for engineers. 6th ed.Wiley;2014.

[11]Mathews P.Design of experiments with MINITAB.ASQ Quality Press;2014.

[12]Stephens MA.EDF statistics for goodness of fit and some comparisons.J Am Stat Assoc 1974:730-7.

[13]Stephens MA.Tests based on EDF statistics.In:D'Agostino RB,Stephens MA, editors.Goodness-of-fit techniques.New York:Marcel Dekker;1986.

[14]Lilliefore HW.On the kolmogorov-smirnov test for normality with mean and variance unknown.J Am Stat Assoc 1967;62:399-402.

[15]Ryan Jr TA,Joiner BL.Normal probability plots and tests for normality. Technical Report,Statistics Department.The Pennsylvania State University; 1976.

[16]Urdan TC.Statistics in plain english.3rd ed.Routledge;2010.

[17]Bonferroni Correction,https://en.wikipedia.org/wiki/Bonferroni_correction [Accessed 11 January 2017].

[18]Erjavec N.Tests for homogeneity ofvariance.In:Internationalencyclopedia of statistical science.Springer;2011.p.1595-6.

[19]Bartlett MS.Properties of suf ficiency and statistical tests.Proc R Soc Lond Ser A 1937;160:268-82.

[20]Brown MB,Forsythe AB.Robust tests for the equality of variance.J Am Stat Assoc 1974;69:364-7.

[21]Levene H.Contributions to probability and statistics.CA,USA:Stanford University Press;1960.

[22]Ochberg Y,Weiss G,Hart S.On graphical procedures for multiple comparisons.J Am Stat Assoc 1982;77(380):767-72.

29 March 2017

*Corresponding author.

E-mail address:daniel.corriveau@drdc-rddc.gc.ca(D.Corriveau).

Peer review under responsibility of China Ordnance Society.

http://dx.doi.org/10.1016/j.dt.2017.04.008

2214-9147/Crown Copyright©2017 Published by Elsevier Ltd.This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/ 4.0/).

Accepted 25 April 2017

Available online 4 May 2017